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ABSTRACT 


This  research  effort  focused  on  the  use  of  Advanced  Warfighting  Experiments 
(AWEs)  to  support  acquisition  decisions.  Specifically,  the  thesis  evaluated  the 
effectiveness  of  the  Army  Task  Force  XXI  AWE  in  providing  information  to  support 
investment  decisions  and  refinement  of  requirements  for  information  age  technologies. 

A  detailed  analysis  of  the  1997  Operational  Test  and  Evaluation  Command 
(OPTEC)  Live  Experiment  Assessment  Report  identified  program  developmental 
recommendations.  Data  were  collected  from  appropriate  program  offices  and  user 
representatives  to  determine  the  perceived  utility  of  the  recommendations  and  level  of 
implementation.  Qualitative  data  detailing  why  specific  recommendations  were  or  were 
not  implemented  were  used  to  determine  the  contributing  factors  to  a  program’s  ability  to 
benefit  from  participation  in  the  experiment. 

Overall,  fifty-two  percent  of  the  OPTEC  recommendations  were  reported  as  either 
fully  or  mostly  implemented.  Other  potential  benefits  of  AWE  participation  were 
identified  to  include  (1)  marketing  and  exposure  of  program,  (2)  refinement  of  user 
requirements,  and  (3)  information  on  integration,  interfaces,  and  interoperabilty.  Risks 
from  participation  in  the  AWE  included  (1)  a  poor  return  on  investment,  (2)  potential 
negative  exposure,  and  (3)  extensive  changes  in  requirements.  Recommendations  to 
enhance  the  value  of  participation  in  AWEs  are  included. 
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I.  INTRODUCTION 


The  principle  of  science,  the  definition,  almost,  is  the  following:  The  test 
of  all  knowledge  is  experiment.  Experiment  is  the  sole  judge  of  scientific 
‘truth.’  But  what  is  the  source  of  knowledge?  Where  do  the  laws  that  are 
to  be  tested  come  from?  Experiment,  itself,  helps  to  produce  these  laws, 
in  the  sense  that  it  gives  us  hints.  But  also  needed  is  imagination  to  create 
from  these  hints  the  great  generalizations — ^to  guess  at  the  wonderful, 
simple,  but  very  strange  patterns  beneath  them  all,  and  then  to  experiment 
to  check  again  whether  we  have  made  the  right  guess.  [Feynman,  1995, 

p2] 

A.  BACKGROUND 

From  March  1996  through  October  1997,  the  Army  conducted  the  Task  Force 
XXI  Advanced  Warfighting  Experiment  (TF  XXI  AWE),  culminating  with  a  live  exercise 
at  the  National  Training  Center,  during  Rotation  97-06,  March  1997.  The  purpose  for 
conducting  the  AWE  was  to  provide  sufficient  data  to  validate  digitizing  the  battlefield 
and  support  credible  assessments  on  which  to  base  future  procurement  decisions.  The  TF 
XXI  AWE  was  meant  to  be  a  tool  for  resolving  issues  and  reducing  risk  early  in  the 
program  development  process  and  determining  the  adequacy  of  requirements,  design,  and 
new  system  capabilities  before  committing  major  resources.  A  total  of  ninety-three  TF 
XXI  new  technology  initiatives  were  initially  included  in  the  AWE. 

The  stated  TF  XXI  AWE  objectives  include  experimenting  with  advanced 
technologies  and  concepts  that  leverage  capabilities  of  information  age  technologies  and 
providing  information  to  support  investment  decisions  on  the  most  promising  initiatives 
[ECC,  1996].  The  AWE  was  also  meant  to  help  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  refine  requirements  and  develop  solutions  for  Force  XXI.  The 
final  Live  Experiment  Assessment  Report,  prepared  by  the  U.S.  Army  Operational  Test 
and  Evaluation  Command,  included  full  assessments  of  most  participating  TF  XXI 
initiatives.  OPTEC  provided  observations  and  specific  developmental  recommendations 
for  each  initiative.  AWE  documentation  does  not  directly  address  the  purpose  and  goals 
of  the  experimental  process  as  it  relates  to  the  acquisition  program  manager.  This  thesis 
attempts  to  partially  define  goals  and  objectives  for  the  program  manager  to  consider 
when  presented  with  the  opportunity  to  participate  in  an  AWE. 
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B.  PURPOSE 


This  thesis  studies  the  use  of  Advanced  Warfighting  Experiments  to  support 
material  acquisition  decisions.  Specifically,  the  thesis  evaluates  the  effectiveness  of  the 
Army  Task  Force  XXI  Advanced  Warfighting  Experiment  (TF  XXI  AWE)  objective  of 
providing  information  to  support  investment  decisions  and  refinement  of  requirements  for 
information  age  technologies.  Research  includes  a  detailed  analysis  of  the  1997 
Operational  Test  and  Evaluation  Command  (OPTEC)  live  Experiment  Assessment 
Report  to  identify  program  developmental  recommendations.  Data  were  collected  from 
appropriate  program  offices  and  user  representatives  to  determine  the  perceived  utility  of 
the  recommendations  and  level  of  implementation.  Subjective  data  detailing  why  specific 
recommendations  were  or  were  not  implemented  were  used  to  determine  the  contributing 
factors  to  a  program’s  ability  to  benefit  from  participation  in  the  experiment.  The  thesis 
includes  recommendations  on  how  to  best  use  Advanced  Warfighting  Experiments  to 
support  the  acquisition  program  manager. 

C.  SCOPE 

The  scope  of  this  thesis  is  limited  in  that  it  concentrates  on  the  1997  Task  Force 
XXI  Advanced  Warfighting  Experiment  and  no  previous  or  subsequent  experiments. 
Only  initiatives  with  detailed  recommendations  provided  by  OPTEC  were  considered. 
The  research  does  not  include  an  analysis  of  the  Joint  Venture  XXI  doctrinal  initiatives  or 
organizational  change  initiatives  but  only  the  developmental  program  recommendations. 
This  thesis  wiU  not  determine  the  advantages  or  disadvantages  of  the  AWE  process 
beyond  the  support  of  acquisition  decisions.  The  results  of  the  research  do  not  determine 
the  overall  success  or  failure  of  the  TF  XXI  AWE.  However,  it  concentrates  on  material 
acquisition  issues,  the  immediate  use  of  OPTEC  recommendations  for  initiative 
development,  and  the  contributing  factors  behind  the  use  or  non-use  of  those 
recommendations. 

D.  RESEARCH  QUESTIONS 

1.  What  were  the  objectives  of  the  Army’s  Task  Force  XXI  Advanced 
Warfighting  Experiment,  what  was  the  structure  of  the  experiment,  and  how  was 
information  gained  from  the  experiment  used  in  making  acquisition  decisions? 
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2.  Were  the  specific  recommendations  derived  from  the  Task  Force  XXI 
Advanced  Warfighting  Experiment  (AWE)  used  to  support  investment  decisions  and  to 
refine  requirements  of  participating  initiatives? 

3.  What  were  the  contributing  factors  to  a  program’s  ability  to  benefit  from 
participation  in  the  Advanced  Warfighting  Experiment? 

4.  What  are  the  characteristics  of  programs  that  are  best  positioned  to  gain  valued 
investment  and  requirements  information  from  participation  in  Advanced  Warfighting 
Experiments? 

E.  METHODOLOGY 

The  methodology  used  in  this  thesis  will  consist  of  the  following  steps. 

1.  Conduct  a  literature  search  of  books,  magazine  articles,  CD-ROM 
systems,  internet  and  other  library  information  resources. 

2.  Conduct  a  thorough  review  of  the  TF  XXI  AWE  Experiment  Plan  and 
OPTEC  live  Experiment  Assessment  Report. 

3.  Creation  of  tailored  surveys  for  each  program  initiative  and  dissemination 
to  each  applicable  program  office  and  user. 

4.  Follow  up  interviews  of  pertinent  program  managers,  users,  and  test 
officials. 

5.  Conduct  a  statistical  analysis  of  collected  data  regarding  implementation 
of  initiative  recommendations. 

6.  Conduct  an  analysis  of  collected  comments  on  contributing  factors  to  a 
program’s  ability  to  benefit  from  participation  in  the  Advanced  Warfighting  Experiment. 

7.  Conduct  an  analysis  and  develop  a  siunmary  description  of  the 
characteristics  of  a  program  that  is  best  positioned  to  participate  in  future  AWEs. 

F.  THESIS  OUTLINE 

I.  Introduction.  This  chapter  provides  information  on  the  scope, 

methodology,  and  purpose  of  the  thesis. 

n.  AWE  Structure  and  Literature  Review.  This  chapter  reviews  the 

background  and  structure  of  the  Army's  experimental  process  and  reviews  the  literature 
addressing  experimental  research  methods. 
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in.  Empirical  Study  Description.  This  chapter  details  the  design  of  the  survey 
instrument  and  the  analysis  strategy. 

IV.  Findings  on  the  Use  of  Derived  Information  in  Supporting  Acquisition 
Decisions.  This  chapter  provides  a  statistical  analysis  and  conclusions  from  gathered  data 
on  the  implementation  of  specific  recommendations  derived  from  the  AWE. 

V.  Analysis  of  Contributing  Factors  to  Benefiting  from  Advanced 
Warfighting  Experiments.  This  chapter  addresses  the  subjective  comments  made  by 
survey  participants  and  interviewees  and  illustrates  lessons  learned  on  the  contributing 
factors  to  benefiting  from  the  AWE.  It  also  describes  the  characteristics  of  a  program  that 
is  best  positioned  to  gain  from  participation  in  future  AWEs. 

VI.  Conclusions  and  Recommendations.  This  chapter  summarizes  lessons 
learned  from  the  thesis  and  makes  recommendations  to  enhance  the  AWE  process  to 
support  the  acquisition  manager. 

G.  EXPECTED  BENEFITS  OF  TfflS  THESIS 

This  thesis  provides  the  acquisition  manager  with  information  necessary  to  make 
decisions  about  participation  in  Advanced  Warfighting  Experiments.  The  thesis  assists 
acquisition  managers  in  gaining  maximum  benefit  from  participation  in  future 
experiments.  The  research  assists  planners  in  tailoring  future  experiments  to  better 
benefit  participating  material  programs. 
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n.  AWE  STRUCTURE  AND  LITERATURE  REVIEW 


A.  THE  TASK  FORCE  XXI  ADVANCED  WARFIGHTING  EXPERIMENT 

1.  Background 

In  February  1995,  the  Chief  of  Staff  of  the  Army  approved  the  Army  Force  XXI 
concept.  The  Force  XXI  concept  served  to  modernize  and  redesign  equipment  and 
procedures  to  meet  the  emerging  twenty-first  century  threat.  Battlefield  digitization  was 
the  central  theme  for  the  envisioned  Force  XXI.  Digitization  includes  the  linking  of  all 
weapon  platforms  and  vehicles  at  the  brigade  level  and  below  into  a  tactical  internet 
capable  of  sharing  a  vast  amount  of  information.  Digitization  was  expected  to  provide 
improved  situational  awareness  and  command  and  control  capabilities.  The  Army 
designed  the  Advanced  Warfighting  Experiment  to  find  the  most  promising  technologies 
to  lead  the  Army  to  Force  XXI.  [Lickteig,  1996] 

The  Army  labeled  its  effort  to  modernize  to  Force  XXI  as  “Joint  Venture.”  Joint 
Venture  was  an  iterative  cycle  of  concept  development,  force  design,  and  experimentation 
[Lickteig,  1996].  The  Army’s  senior  leadership  described  the  Joint  Venture  Campaign 
Plan  as  “a  comprehensive  approach  to  redesign  the  force — organized  around 
information — to  be  inherently  more  versatile  and  flexible”  [Lickteig,  1996,  p3].  Through 
the  initial  efforts  of  Joint  Venture,  the  Army  discovered  it  needed  a  more  pragmatic  and 
responsive  research  method  that  would  clearly  provide  relevant  and  credible  analysis  to 
validate  the  premise  of  Force  XXI  [TRADOC,  1996].  The  Army  developed  the 
Advanced  Warfighting  Experiments  (AWE)  as  the  focal  effort  in  establishing  Force  XXI. 
The  AWEs  were  based  on  “an  iterative  sequence  and  mix  of  warfighting  simulations — 
live,  constructive  and  virtual — ^in  which  soldiers  and  units  conduct  realistic  tactical 
operations”  [Lickteig,  1996,  p6]. 

The  Task  Force  XXI  AWE  was  one  of  a  series  of  AWEs  designed  to  implement 
the  Joint  Venture  plan.  The  Joint  Venture  Experimentation  Plan  is  detailed  in  Figure  1 
and  illustrates  the  path  from  the  current  brigade,  division,  and  corps  design  to  the 
modernized  Force  XXI.  The  AWE  series  began  with  the  Focused  Dispatch  (FD)  AWE. 
Focused  Dispatch  consisted  of  a  heavy  battalion  task  force  conducting  simulated 
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exercises  from  February  1995  through  August  1995.  FD  AWE  focused  on  developing 
digitized  training  support  packages  and  validating  doctrine  and  techniques  for  digitizing 
heavy  forces.  The  FD  A'WE  was  followed  by  the  Warrior  Focus  (WF)  AW^.  Warrior 
Focus  was  conducted  from  May  1995  through  November  1995  using  a  light  infantry 
brigade  and  culminated  with  a  training  exercise  at  the  Joint  Readiness  Training  Center 
(JRTC)  at  Fort  Polk,  Louisiana.  The  WF  AWE  assisted  in  the  Army  developing  tactics, 
techniques,  and  procedures  (TTPs)  in  operating  and  equipping  a  digitized  light  infantry 
battalion.  The  TF  XXI  AWE  was  followed  by  the  Division  (DIV)  AWE  held  in 
November  1997.  The  Army  designed  the  DIV  XXI  AWE  to  provide  insights  on  echelons 
above  division  andjoint  digitized  operations.  [ECC,  1996] 


flOmiGaASEUUBE 


Figure  1.  Joint  Venture  Experimentation  Plan  [PM  FBCB2, 1995] 

2.  Purpose  and  Goals 

The  central  hypothesis  of  the  TF  XXI  AWHE  was  developed  by  the  Army’s 
Training  and  Doctrine  Command  and  states  "that  if  information-age  battle  command 
capabilities  and  connectivity  exists  across  all  battlefield  operating  system  (BOS) 
functions  in  a  brigade  task  force,  then  increases  in  lethality,  survivability,  and  tempo  will 
be  achieved"  [ECC,  1996,  l.b.l].  The  overall  objective  of  the  AWE  was  to  "focus 
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modernization  efforts  on  the  most  promising  technologies.. .that  have  the  greatest 
potential  for  enhancing  force  capabilities"  [ECC,  1996,  l.c.l]. 

The  Experiment  Directive  published  five  specific  objectives  for  the  AWE: 

a.  Assess  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2) 
Applique  and  Tactical  Internet  capabilities  to  refine  FBCB2  requirements. 

b.  Assess  digitized  brigade  combat  service  support  concept. 

c.  Experiment  with  advanced  technologies  and  concepts  that  leverage 
capabilities  of  information  age  technologies. 

d.  Refine  digitized  tactics,  techniques,  and  procedures  (TTP)  for 
brigade  operations. 

e.  Provide  information  to  support  investment  decisions  on  the  most 
promising  initiatives.  [ECC,  1996] 

Fundamental  to  the  Army’s  process  of  Joint  Venture  is  the  concept  that 
development  involves  “continuous  experimentation,  discovery  learning,  and  iterative 
refinement"  [Lickteig,  1996,  p4].  The  rapid  advance  of  technology  has  resulted  in  an 
inherent  inability  for  Army  leadership  to  predict  the  eventual  composition  and 
organization  of  its  force.  Information  technology  is  providing  an  unprecedented  level  of 
capability  to  provide  a  common  picture  of  the  battlefield  situation  to  all  participants.  The 
defining  purpose  of  the  AWE  is  to  provide  a  forum  to  implement  continued,  iterative 
improvements. 

AWE  documentation  does  not  directly  address  the  purpose  and  goals  of  the 
experimental  process  as  it  relates  to  the  acquisition  program  manager.  This  thesis 
attempts  to  partially  define  goals  and  objectives  for  the  program  manager  to  consider 
when  presented  with  the  opportunity  to  participate  in  an  AWE. 

3.  AWE  Scope  and  Methodology 

a.  Scope 

The  TF  XXI  AWE  initially  consisted  of  thirty-six  Joint  Venture  issues  and 
niilety-two  new  technology  initiatives.  The  Joint  Venture  issues  were  proposed  research 
questions  regarding  organizational  doctrine,  tactics,  procedures,  leader  development, 
organization,  and  information  management.  The  new  technology  initiatives  included 
advanced  technology  products  in  various  stages  of  development  [ECC,  1996].  Officials 
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reduced  the  number  of  participating  technology  initiatives  prior  to  the  experiment  based 
on  the  ability  of  the  initiative  to  contribute  to  the  overall  goals  of  the  experiment.  The 
experiment  plan  focused  on  the  initiatives  central  to  digitization,  such  as  the  Applique 
(now  titled  Force  XXI  Battle  Command  Brigade  and  Below  -  FBCB2)  and  those 
initiatives  that  contributed  to  the  full  exercise  of  these  central  programs.  This  thesis 
concentrates  on  these  initiatives  and  does  not  conduct  an  analysis  of  the  value  obtained 
from  study  of  the  Joint  Venture  issues. 

The  live  exercise  consisted  of  platoon,  company,  battalion,  and  brigade 
sized  units  conducting  train-up  and  simulated  combat  operations  against  an  aggressive 
opposing  force.  The  experiment  used  pre-digitization  exercises  to  develop  a  baseline  to 
measure  the  effects  of  the  digital  enhancements  and  the  resulting  increases  in  unit 
effectiveness.  The  TF  XXI  AWE  was  designed  to  measure  the  holistic  effect  of 
synchronizing  all  elements  of  a  combined  arms  team  and  to  replicate  “the  same  free  flow 
operations  and  decisions  that  would  be  expected  on  the  battlefield”  [ECC,  1996,  l.c.8]. 

b.  Initiative  Selection  Procedures 

TRADOC  selected  initiatives  to  participate  in  the  TF  XXI  AWE  based  on 
their  ability  to  contribute  to  the  objectives  of  the  exercise.  Some  initiatives  were  logically 
included  because  they  formed  the  basic  infrastructure  of  the  Army’s  overall  digitization 
plan,  such  as  Applique/FBCB2  and  the  Tactical  Internet.  TRADOC  selected  other 
systems  because  of  their  ability  to  contribute  to  the  data  flow  required  to  exercise  the 
digitization  backbone,  such  as  the  Longbow  Apache  Helicopter  and  the  Linebacker  Air 
Defense  Artillery  system.  Emerging  technologies  that  were  new  concepts  to  the  Army  in 
terms  of  doctrinal  employment  and  capabilities  determination  formed  a  third  group,  such 
as  the  Stingray  Combat  Protection  System  and  the  Lightweight  Video  Reconnaissance 
System  (LVRS).  Finally,  even  though  they  did  not  directly  support  the  goals  of  the 
experiment,  a  fourth  group  of  initiatives  were  included  because  of  their  status  as 
emerging  technologies.  It  is  unclear  how  formal  and  rigorous  the  selection  process  was 
for  the  TF  XXI  AWE.  However,  from  the  results  and  lessons  learned  from  the  AWE,  the 
Army  has  developed  a  structured  system  for  initiative  selection  in  future  AWEs  that  is 
designed  to  insure  that  each  system  constructively  supports  the  goals  and  objectives  of  the 
experiment. 
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The  formal  issue  and  initiative  submission  and  review  process  conducted 
by  TRADOC  for  the  1999  Joint  Contingency  Force  AWE  includes  four  phases:  (1) 
submission  of  proposed  issues  to  the  TRADOC  Analysis  Center  (TRAC)  and  the 
appropriate  battle  lab,  (2)  consolidation  and  initial  review  of  issues  and  initiatives  at  the 
TRADOC  level,  (3)  approval  of  issues  and  initiatives  by  the  selection  committee,  and  (4) 
the  continuous  monitoring  and  refinement  of  the  approved  issues  and  initiatives.  The 
main  criteria  for  selection  include  four  components:  (1)  the  initiative’s  relationship  to  an 
AWE  hypothesis  or  objective,  (2)  the  availability  of  the  initiative  by  the  designated 
“Everything-in-Place  Date”,  (3)  the  availability  of  the  initiative  to  be  integrated  into  the 
architecture  of  the  AWE,  and  (4)  the  ability  of  the  submitting  agency  to  bear  the  complete 
costs  associated  with  participation.  The  submitting  agency  must  also  explain  the  impact 
of  the  issue  or  initiative  on  the  Army’s  DOTMPL  framework.  The  DOTMPL  framework 
is  a  stractured  method  the  Army  uses  to  discuss  the  effects  of  new  concepts  in  terms 
doctrine,  organization,  training,  materials,  people,  and  leadership.  [TRADOC  JCW- 
AWE,  1998] 

c.  Data  Collection 

During  the  1997  TF  XXI  AWE,  automated  instrumentation  collected 
digital  data  while  audio  recording  equipment  was  used  to  capture  voice  message  traffic  on 
FM  radio  nets.  Real  time  casualty  assessment  (RTCA)  instrumentation  collected  data  on 
force  effectiveness  and  movement  rates.  Simulated  Area  Weapons  Effects-Multiple 
Integrated  Laser  Engagement  system  (SAWE-ME^SII)  provided  results  of  unit 
engagements.  Most  significantly.  Subject  Matter  Experts  (SMEs)  provided  direct  input  to 
qualitatively  measure  initiative  contribution  to  force  effectiveness.  The  use  of  SMEs  was 
essential  in  understanding  the  complex  interactions  between  initiatives  [OPTEC,  1997]. 
The  primary  emphasis  for  the  experiment  were  those  initiatives  that  directly  contributed 
to  situational  awareness  improvements.  Most  other  technological  initiatives  were 
considered  of  secondary  importance  and.  were  not  measured  quantitatively.  These 
individual  system  contributions  were  not  analyzed  individually  but  as  part  of  the  total 
digitization  package  [ECC,  1996]. 
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d.  Analysis  Plan 

Training  and  Doctrine  Command  Analysis  Center  -  White  Sands  Missile 
Range  (TRAC-WSMR),  the  Operational  Evaluation  Command  (OEC),  and  the  Test  and 
Experimentation  Command  (TEXCOM)  jointly  developed  the  analysis  plan  for  the  TF 
XXI  AWE.  The  analysis  plan  detailed  issues,  measures  of  effectiveness  (MOEs),  and 
analysis  techniques  to  be  used  during  the  live  simulation  [ECC,  1996].  TEXCOM’ s 
representative  for  the  live  portion  of  the  experiment  was  the  Army’s  Operational  Test  and 
Evaluation  Command  (OPTEC).  The  analysis  team,  combined  with  the  Army  Battle 
Labs,  invested  a  significant  effort  to  complete  a  set  of  coordinated,  finalized  initiative 
assessment  plans  [OPTEC,  1997]. 

4.  Limitations 

The  planning  documentation  for  the  TF  XXI  AWE  acknowledges  the  inherent 
limitations  to  the  Army’s  study.  ‘The  planned  TF  XXI  experiments  do  not  follow  the 
formal  study  process,  the  traditional  test  and  evaluation  process,  nor  do  they  follow  the 
strict  procedures  of  a  laboratory  experiment”  [ECC,  1996,  l.c.16].  The  attempt  to 
combine  training  with  assessment  of  digitization  does  not  allow  a  controlled,  repeatable 
experiment.  Because  of  the  training  focus  of  the  exercise,  data  collection  efforts  must  be 
unobtrusive.  This  requirement  limits  the  control  of  factors  and  conditions  required  to 
support  analysis  [ECC,  1996]. 

a.  Repeatability 

The  free-play  nature  of  the  experiment  hmited  the  ability  to  repeat  the 
results  of  any  particular  scenario.  The  sheer  number  of  variables  makes  this  tasks 
infeasible.  Data  collection  failures  due  to  instrumentation  or  equipment  malfunctions 
were  identified  risk  areas  that  could  not  be  mitigated. 

b.  Data  Collection 

Data  collection  on  all  systems  besides  the  Applique,  tactical  internet,  and 
electronic  interfaces  was  limited  to  SME  observations  and  user  surveys.  The  experiment 
directive  spoke  directly  to  the  data  collection  limitations.  “The  majority  of  the  initiatives 
will  not  have  objective  data  to  support  milestone  decisions.  The  SME  and  user  data  on 
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these  second  priority  initiatives  will  be  most  useful  in  refining  requirement  documents 
and  highlighting  potential  strengths  and  weaknesses”  [ECC,  1996,  l.c.16].  Lessons 
learned  from  previous  AWEs  demonstrated  to  the  TF  XXI  planners  that  SME  insights 
and  observations  were  the  most  meaningful  in  terms  of  reliability  and  significance. 
“Attempting  to  isolate  the  individual  contributions  of  each  experimental  system  to  overall 
unit  performance  is  impossible”  [ECC,  1996, 2.b.l.a]. 

The  non-intrusive  data  collection  required  of  an  experiment  overlaid  on 
training  significantly  limited  the  ability  to  gather  meaningful  information.  Data  collection 
over  its  entire  process  lifecycle  from  conception  to  termination  requires  a  resource 
commitment  that  is  infeasible  in  a  free-play  environment.  The  effects  of  specific  actions 
and  their  ramifications  on  other  variables  could  not  adequately  be  measured  with  the 
limitations  of  the  experiment  [OPTEC,  1997]. 

c.  Interactions 

The  large  number  of  interacting  initiatives  participating  in  the  AWE 
combined  with  uncontrolled  and  random  variations  lead  to  hopelessly  confounded  results 
that  could  not  be  isolated  to  demonstrate  respective  contributions.  In  true  laboratory 
experiments,  variables  are  modified  individually  or  according  to  specific  patterns  that 
provide  an  ability  to  decipher  impacts  of  variation.  The  nature  of  an  experiment  overlaid 
upon  a  training  exercise  prevents  the  direct  control  required  to  limit  corruption  of  data  by 
variable  interaction.  The  large  effects  of  interactions  cause  the  subjective  SME 
observations  to  represent  a  significant  level  of  importance. 

d.  Small  Sample  Size 

The  relatively  small  sample  of  focused  iterations  during  the  live 
experiment  coupled  with  the  large  number  of  initiatives  involved  provide  for  statistically 
insignificant  results.  Sufficient  time  and  resources  are  not  available  during  the  AWE  to 
develop  statistically  meaningful  data.  This  small  sample  sizes  make  it  very  difficult  to 
“prove”  hypotheses  as  tme.  [ECC,  1996].  The  RAND  corporation  has  years  of 
experience  conducting  research  at  the  National  Training  Center  and  has  significant 
expertise  in  conducting  effective  quantitative  and  qualitative  experiments.  The  National 
Training  Center  conducts  monthly  "rotations"  lasting  14  days,  training  a  new  brigade¬ 
sized  unit  each  month.  The  TF  XXI  AWE  consisted  of  one  single  NTC  rotation.  RAND 
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admits  that  the  single-rotation  AWE  concept  is  limited  in  what  quantitative  results  it  can 
provide.  RAND  has  found  that  at  least  twelve  rotations’  worth  of  data  is  required  to 
provide  statistically  meaningful  results.  RAND  does  go  on  to  state  that  AWEs  can 
readily  provide  significant  qualitative  results.  [Grossman,  1995]. 

e.  System  Immaturity 

As  stated  in  the  experiment  directive,  some  new  systems  participating  in 
the  experiment  were  immature,  surrogate  in  nature,  and  did  not  have  the  capability  to 
demonstrate  actual  or  desired  performance  [ECC,  1996]. 

5.  Results 

Official  press  releases  after  completion  of  the  AWE  labeled  the  exercise  nearly  a 
complete  success.  The  commanding  officer  of  TRADOC,  General  William  W.  Hartzog, 
stated  that  about  eighty-five  percent  of  the  prototypes  performed  to  requirements.  Of  the 
rest,  ten  percent  needed  improvements  to  be  usefiil  and  five  percent  were  “ideas  whose 
times  have  not  yet  come”  [TRADOC  PAO,  1997]. 

The  primary  feedback  mechanism  for  the  AWE  was  Operational  Test  and 
Evaluation  Command’s  Live  Assessment  Report.  The  report  provided  summations  for 
most  participating  initiatives  as  well  as  specific  recommendations  for  improvements  in 
technology.  Additionally,  the  assessment  report  summarized  the  achievements  of  the 
AWE. 

a.  Better  definition  and  understanding  of  future  requirements. 

b.  Early  identification  of  systems  having  potentially  high  payoff 
capabilities,  as  well  as  systems  needing  significant  developmental  work. 

c.  hidependent  assessment  to  the  Army’s  senior  leadership. 

d.  Early  feedback  to  program  managers. 

e.  Acceleration  of  the  normal  acquisition  process  by  placing 
prototype  systems  into  the  hands  of  soldiers.  [OPTEC,  1997] 

In  its  final  report,  OPTEC  also  identified  significant  lessons  learned  from  the 
AWE  process.  While  OPTEC  recognized  that  the  AWE  offered  excellent  opportunities 
for  the  identification  and  refinement  of  system  requirements,  the  combination  of 
experiment  and  training  does  not  support  operational  testing  requirements.  In  order  to 
gain  beneficial  test  data,  training  effectiveness  would  be  reduced.  However,  OPTEC  did 
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acknowledge  that  the  AWE  was  an  excellent  opportunity  to  identify  critical  operational 
issues  and  criteria  (COIC)  that  would  be  beneficial  for  use  in  future  formal  tests  and 
evaluations  [OPTEC,  1997]. 

B.  DEFINING  THE  PROCESS:  TRAINING,  TESTING,  ASSESSMENT, 
DEMONSTRATION,  OR  EXPERIMENT 

1.  Overview 

The  Army  leadership  has  spent  considerable  effort  in  defining  the  task  and 
purpose  of  their  experimentation.  Planners  initially  chose  the  word  “experiment”  to 
differentiate  the  exercise  from  two  different  categories  of  activities  that  had  already  been 
labeled  “test”  and  “demonstration."  A  presentation  for  a  past  AWE  stated  that  leadership 
should  “reinforce  experiment  versus  demonstration  versus  test”  [Lickteig,  1996].  The 
Advanced  Warfighting  Experiment  had  a  central  hypothesis  and  attempted  to  apply  a 
rudimentary  scientific  method  but  was  criticized,  as  described  in  the  next  section  of  this 
thesis,  as  not  being  a  true  scientific  study. 

Concepts  and  terms  used  in  the  development  of  an  experiment  or  evaluation  plan 
often  have  a  “richness  of  meaning”  [Babbie,  1990,  pi  19].  That  is,  they  contain  a  variety 
of  elements  that  attempt  to  summarize  complex  phenomenon.  To  permit  rigorous 
research,  such  general  concepts  must  be  reduced  to  specific,  empirical  indicators.  These 
indicators  must  be  reduced  to  simplified,  precisely  defined  terms,  hi  the  case  where 
concepts  cannot  be  precisely  defined,  the  outcomes  cannot  be  measured  precisely.  A 
tenet  of  scientific  research  is  that  data  are  not  collected  but  rather  created.  The  creation  of 
data  is  highly  dependent  on  the  definition  of  variables  and  attributes  [Babbie,  1993]. 

The  AWE  experiment  directive  used  terms  such  as  “gathering  information”, 
“making  decisions”,  “gaining  insight”,  “evaluation”,  “validation”  and  “assessment” 
without  clearly  defining  the  application  of  the  terms.  It  is  unclear  how  the  Army  initially 
applied  the  term  experiment  and  how  experimentation  would  interact  with  the  other 
developmental  activities  already  in  place.  The  AWE  analysis  plan  also  identified  that 
there  would  be  a  wide  range  of  analytical  rigor  in  the  assessment  of  initiatives.  A  final 
complication  introduced  by  the  designers  of  the  AWE  was  the  overlay  of  the  event  on  a 
pre-established  training  exercise  at  the  National  Training  Center  (NTC),  Fort  Irwin, 
California.  The  NTC  is  commonly  referred  to  as  the  Army’s  most  highly  rated  training 
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opportunity  for  mechanized  forces.  The  description  of  the  AWE  as  a  training  event 
additionally  confounded  the  purpose  of  the  exercise. 

Through  the  experience  gained  in  the  TF  XXI  AWE,  following  exercises,  and 
detailed  studies  of  the  Army’s  experimentation  process,  the  Army  has  more  clearly 
defined  its  experimental  method,  the  terms  used  in  its  analysis  plans,  and  the  interaction 
experimentation  plays  with  other  developmental  activities.  The  Army  has  determined 
that  future  exercises  must  be  based  on  sound,  integrated,  all-inclusive  analysis 
methodology. 

2.  Training 

The  Army  defines  training  as  “the  instruction  of  personnel  to  individually  and 
collectively  increase  their  capability  to  perform  specific  functions  or  tasks”  [CALL, 
1999].  OPTEC  stated  in  the  TF  XXI  AWE  Live  Simulation  Assessment  Plan  that  “the 
primary  objective  of  the  exercise  is  training”  [ECC,  1996,  Annex  C,  Appendix  B,  para. 
2.3.1].  The  Army’s  National  Training  Center  was  founded  with  the  purpose  of  providing 
the  most  realistic  training  exercises  possible  short  of  actual  war.  Secondarily,  the  NTC 
was  designed  to  provide  “lessons  learned”  for  the  Army. 

The  NTCs  training  mission  emphasis  has  a  significant  implication  on  its  ability  to 
serve  as  an  environment  for  an  Advanced  Warfighting  Experiment.  Although  NTC 
rotations  approximate  actual  combat,  they  do  not  exactly  replicate  it.  Even  with  extensive 
improvements  in  technological  capability,  the  environment  of  the  NTC  is  designed  such 
that  it  is  unlikely  that  the  evaluation  force  will  achieve  victory  in  any  one  engagement. 
The  NTC  also  focuses  on  certain  missions  and  battlefield  operating  systems.  Those 
initiatives  not  realistically  exercised  in  the  training  environment  should  not  be  included  in 
the  experiment.  The  situation  has  added  complexity  in  that  training  is  combined  with  the 
introduction  of  new  equipment.  It  is  difficult  to  separate  problems  related  to  training  and 
problems  related  to  new  initiatives.  Actual  effects  of  applied  initiatives  may  also  be 
limited  due  to  the  need  to  continue  the  exercise  in  a  proper  force  correlation  to  serve  the 
training  objective.  [Grossman,  1995] 

3.  Testing 

Testing  in  the  Army  is  divided  between  developmental  and  operational  activities. 
Both  are  designed  to  be  assessments  of  an  item's  ability  to  meet  designed  requirements 
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and  parameters.  Operational  testing  is  an  official  requirement  to  conduct  independent 
analysis  of  equipment  prior  to  production.  While  the  live  simulation  assessment  plan 
included  in  the  experiment  directive  a  dendritic  of  functional  issues,  measures  of 
effectiveness  (MOE),  measures  of  performance  (MOP),  and  data  sources  as  normally  seen 
in  an  operational  test  and  evaluation  plan,  representatives  from  the  Army’s  Test  and 
Evaluation  Command  were  adamant  that  the  exercise  was  not  a  “test”  and  would  not 
replace  proper  operational  test  and  evaluation  of  any  initiative. 

The  live  simulation  assessment  plan  stated  that  the  planned  Task  Force  XXI 
experiments  would  not  follow  the  traditional  operational  test  and  evaluation  process. 
Because  of  the  severe  limitations  imposed  by  the  nature  of  the  experiment  combined  with 
the  needs  of  the  training  exercise,  any  data  collected  would  be  insufficient  in  drawing  any 
conclusions  required  for  a  true  operational  test  [ECC,  1996].  The  NTC  is  not  designed  as 
a  test  and  evaluation  center.  Although  the  NTC  observer/controllers  are  extremely 
beneficial  in  the  conduct  of  AWEs,  they  will  not  comprondse  their  training  function 
[Grossman,  1995].  In  a  personal  interview,  a  representative  from  the  Headquarters  of 
OPTEC  stated  that  while  the  AWE  did  not  count  as  operational  testing  for  any 
developmental  item,  it  was  included  in  the  spectrum  of  “continuous  evaluation.”  He  went 
on  to  state  that  developmental  and  operational  testing  are  merging  and  that  if  proper 
actions  are  taken  by  the  developmental  community  after  the  AWE,  that  the  system  might 
require  less  testing  and  scrutiny  when  the  time  for  operational  testing  arrives. 

4.  Assessment 

The  Army  defines  assessment  as  “the  determination  of  the  overall  effectiveness” 
[CALL,  1999].  The  term  “assessment”  was  not  clearly  defined  in  the  TF  XXI  experiment 
directive.  However,  future  AWE  frameworks  clearly  define  the  term  “assess”  as  “to 
analyze  critically  and  judge  definitely,  the  nature  and  merit  of  mature  systems, 
technologies,  and  procedures”  [TRADOC  JCF-AWE,  1998,  A-1].  The  analytical  rigor  of 
the  term  “assess”  is  considered  very  high. 

Assessment  plans  developed  for  the  TF  XXI  AWE  were  a  result  of  “a  collective 
effort  between  the  proponent  and  OPTEC”  [ECC,  1996,  Appendix  B,  Annex  1]. 
Initiative  proponents  provided  draft  assessment  plans  to  OPTEC.  OPTEC  revised  the 
plans  to  match  the  available  collection  assets  and  also  added  issues  of  their  own. 
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Proponents  for  initiatives  were  normally  the  specific  Battle  Lab  that  had  responsibility  for 
that  particular  area  of  interest 

5.  Demonstrations 

The  term  “demonstration”  was  clearly  defined  for  experiments  following  the  TF 
XXI  AWE  as  a  hierarchical  experimental  term  that  recognizes  the  level  of  maturity  of  a 
system..  “Demonstrate  &  Deploy”  was  defined  as  “to  show  by  virtual,  simulated,  or 
actual  operation  a  potential  capability  for  prototype  systems^  technologies,  and 
procedures”  [TRADOC  JCF-AWE,  1998,  A-1].  The  analytical  rigor  for  demonstrations 
is  considered  high. 

The  Army  conducts  two  different  demonstration  programs.  Advanced  Technology 
Demonstrations  (ATDs)  and  Advanced  Concept  Technology  Demonstrations  (ACTDs). 
The  primary  objective  of  the  ACTD  process  is  to  “accelerate  and  facilitate  application  of 
mature  advanced  technologies  to  solve  important  military  problems”  [USAMRMC,  1997, 
pi].  ACTDs  conduct  realistic  and  extensive  military  exercises  to  provide  the  user  an 
opportunity  to  evaluate  utility  and  gain  experience  with  mature  technologies.  ACTDs  are 
funded  by  participating  initiatives,  supplemented  by  funding  from  the  ACTD  funding 
line.  ATDs  demonstrate  the  potential  for  enhanced  operational  capability  and  help  to 
speed  the  maturation  of  advanced  technologies.  They  are  funded  by  initiative  Research, 
Development,  Test,  and  Evaluation  (RDT&E)  funds.  [USAMRMC,  1997] 

The  need  to  properly  label  the  activities  that  would  become  Advanced 
Warfighting  Experiments  as  something  separate  from  demonstrations  was  evident  early  in 
the  program  development  process.  General  Gordon  Sullivan,  the  architect  for  Joint 
Venture  stated  “first  we  called  them  demonstrations,  and  that  didn’t  fly  because  we  knew 
if  it  was  a  demonstration,  that  wouldn’t  be  satisfactory  for  any  of  us.  It  would  be  an 
experiment”  [Lickteig,  1996]. 

6.  Experiments 

Experiment  connotes  discovery  learning.  Experiments  are  “formative  exercises  to 
see  what  works  and  what  doesn’t”  [Lickteig,  1996,  pl5].  The  strict  application  of  the 
scientific  method  to  experimentation  is  a  precisely  controlled  research  method.  The 
original  term  of  Advanced  Warfighting  Demonstrations  was  changed  to  Advanced 
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Warfighting  Experiments  to  show  the  application  of  more  realistic  conditions  that  tasked 
soldiers  and  equipment  to  wartime  levels.  [Lickteig,  1996] 

The  Army’s  Advanced  Warfighting  Experiment  appears  to  be  based  on  a  strictly 
scientific  model.  The  Army  has  developed  hypotheses  and  emphasized  the  validation  of 
models  against  baselines.  The  Army  has  also  stated  in  briefings  that  “the  scientific 
method  paradigm  must  be  adapted”  [Lichteig,  1996,  p8]. 

The  U.S.  Army  Research  Institute  recommended  that  the  Army  implement  a 
formative  evaluation  method  that  focuses  on  exploration,  explanation,  and  improvement. 
The  Army  has  developed  an  analytic  method  it  refers  to  as  Model-Experiment-Model- 
Validate  (MEMV).  Models  are  initially  developed  by  battle  labs  and  experimented 
through  simulation.  The  models  are  then  adjusted  based  on  derived  experimental  data 
and  validated  through  live  exercises.  The  role  of  the  AWE  in  the  Army’s  experimental 
model  is  discussed  in  the  following  section.  [Lickteig,  1996] 

Advanced  Warfighting  Experiments  are  not  the  Army’s  only  experimental 
activity.  The  Army’s  Battle  Labs  also  conduct  their  own  warfighting  experiments.  The 
Battle  Lab  Warfighting  Experiments  (BLWE)  are  smaller,  more  focused  assessments  of  a 
single  battle  dynamic.  [Lickteig,  1996] 

7.  Other  Key  Definitions 

a.  Validate 

The  term  validate  has  been  defined  for  future  AWEs  to  mean  “confirm”  or 
“substantiate”.  “It  is  not  used  in  the  strict  experimental  sense  where  there  is  a  comparison 
of  alternatives”  [TRADOC,  1996, 2.e]. 

b.  Gain  Insights 

The  term  “gain  insights”  has  been  defined  for  future  AWEs  as  an 
experimental  term  that  recognizes  the  level  of  maturity  for  a  concept.  The  term  “gain 
insights”  is  defined  “to  obtain  a  clear  and  immediate  understanding  of  emerging  concepts, 
systems,  technologies,  and  procedures”  [TRADOC  JCF-AWE,  1998,  A-1]. 
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c.  Objectives 


Objectives  are  specific,  operational  statements  detailing  the  desired 
accomplishments  of  a  program.  The  planning  and  implementation  of  programs  are 
predicated  on  the  need  to  reduce  the  gap  between  objectives  and  reality.  Effectively 
stated  objectives  use  strong  verbs,  state  only  one  purpose  or  aim,  specify  a  single  end- 
product  or  result,  and  specify  the  expected  time  for  achievement.  [Rossi,  1993] 

C.  EXPERIMENTAL  RESEARCH  METHODS 

“Scientific  research  has  two  primary  goals:  description  and  explanation. 
Researchers  measure  the  empirical  distribution  of  values  on  variables  (description)  and 
the  associations  between  variables  for  purposes  of  explaining  the  distribution  of  values” 
[Rossi,  1993].  A  wide  spectrum  exists  on  the  application  of  a  strict  scientific  model  on 
warfighting  experiments.  The  U.S.  Army  Research  Institute  believes  that  research 
objectives  should  determine  research  methods  [lickteig,  1996].  The  complex  nature  of 
the  AWE  may  not  be  facilitated  by  a  strict  experimental  model.  Basic  research  theory 
demonstrates  that  proving  a  true  difference  among  two  states  is  much  more  difficult  than 
proving  no  difference  [Lickteig,  1996].  AWEs  will  not  benefit  from  lessons  learned  in  a 
“no  difference  found”  environment.  The  Army  has  admitted  that  it  is  “experimenting 
about  experimenting”  [Lickteig,  1996,  pi 5], 

1.  The  AWE  Application  of  the  Scientific  Method 

The  scientific  method  of  experimentation  is  a  precisely  controlled  research 
method.  Scientists  experiment  by  randomly  assigning  participants  to  an  experimental  and 
control  group  and  strictly  control  factors  “extraneous  to  the  causal  variable  of  interest” 
[Lickteig,  1996,  pl5].  Multiple  studies  have  criticized  the  Army’s  use  of  the  scientific 
method  in  experimentation  and  recommended  more  structured  and  applicable  methods. 
The  AWEs  have  neither  random  assignment  of  participants  nor  strict  control  over 
variables.  For  the  TF  XXI  AWE,  the  experimental  and  control  group  were  the  same, 
equipped  differently.  The  wide  range  of  variability  in  the  exercise  fuels  debate  on  the 
validity  of  any  of  the  conclusions  reached. 
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According  to  a  study  done  by  the  U.S.  Army  Research  Institute,  the  Army  has 
recognized  the  limitations  in  its  implementation  of  the  scientific  method  but  may  not  have 
adequately  addressed  them. 

Rather  than  empirical  control  over  conditional  differences,  or  a  systematic 
design  to  reduce  variation,  the  plan  proposes  to  document  such  differences 
(e.g.,  issues,  design,  structure,  equipment)  and  conditions  unique  to  each 
exercise,  in  a  relational  data  base.  The  documentation  of  all  exercise 
conditions  is  essential  for  interpretation  of  findings.  However,  it  is  no 
substitute  for  the  controls  required  to  establish  commensurate  conditions, 
systematic  changes  in  treatment,  and  definitive  or  validated  outcome 
improvements.  [Lickteig,  1996,  pi 6] 


The  Research  Institute  proposes  that  AWEs  do  not  prevent  designers  from 
controlling  strict  controls  on  variables.  The  necessary  control  might  be  achieved  by 
dedicating  a  portion  of  the  experiment  to  small,  detailed  sub-experiments.  The  results  of 
the  scientific  study  should  also  focus  on  providing  solutions  rather  then  identifying 
failure.  [Lickteig,  1996] 

2.  Formative  Versus  Summative  Evaluations 

The  U.S.  Army  Research  Institute  published  its  findings  on  appropriate  research 
methods  for  Advanced  Warfighting  Experiments  and  believes  that  the  overarching 
research  strategy  should  consist  of  a  balance  between  formative  and  summative 
evaluations  [Lickteig,  1996].  Formative  and  summative  evaluations  differ  in  their  focus 
and  role. 

Formative  studies  are  defined  as  “evaluative  activities  undertaken  during  the 
design  and  pretesting  of  programs  to  guide  the  design  process  [Rossi,  1993,  pi 04].  These 
formative  studies  may  be  simple  or  “as  complex  and  comprehensive  as  full-blown 
evaluations”  [Rossi  &  Freeman,  1993].  Formative  evaluations  focus  on  intermediate 
goals  and  play  a  productive  role  in  their  achievement.  Formative  evaluations  are 
conducted  in  developmental  stages  to  help  form  or  improve  the  system  for  the  user 
[Lickteig,  1996]. 

Summative  evaluations  address  final  goals  as  opposed  to  the  intermediate  goals  of 
formative  evaluations.  Summative  evaluations  address  the  success  or  failure  of  goal 
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achievement  as  opposed  to  taking  an  active  roll  in  their  achievement.  Summative 
evaluations  generally  provide  information  for  an  external  audience.  [Lickteig,  1996] 

While  both  formative  and  summative  evaluations  can  be  conducted  during  AWEs, 
an  emphasis  on  formative  issues  may  avoid  a  complete  conclusion  of  failure  driven  by 
summative  evaluations.  The  macro-level  focus  of  AWEs  can  better  be  served  by  the 
formative  evaluation  model.  Summative  evaluations  require  exacting  methods  that 
cannot  be  facilitated  in  the  free-play,  training  environment  of  the  AWE.  The  Joint 
Venture  concept  of  iterative,  continual  improvement  also  models  formative  evaluations 
more  directly.  Since  Joint  Venture  is  formative  in  nature,  the  belief  that  AWEs  cannot 
satisfy  the  requirements  for  operational  test  and  evaluation  is  supported  due  to  the 
summative  nature  required  for  operational  testing.  Finally,  the  premature  focus  on 
summative  results  of  individual  initiatives  might  lead  to  the  faulty  conclusion  that 
advanced  technology  devices  are  not  beneficial  to  the  force  as  a  whole.  [Lickteig,  1996] 

3.  The  Credible  Uses  Framework 

In  1998,  the  RAND  Corporation  published  its  findings  on  the  application  of  the 
“Credible  Uses”  (CU)  framework  to  Advanced  Warfighting  Experiments  [Lucas,  Banks, 
Vye,  1998]. 

The  CU  framework,  originally  documented  by  James  A.  Dewar  (et  al.)  [Dewar,  et 
al.l996],  is  based  on  a  decision-to-experiments  ladder  (DEL)  that  directly  links 
experiments  to  decisions  [Lucas,  Moore,  Vye,  1998].  Advanced  Warfighting 
Experiments  are  limited  in  their  ability  to  predict  real  world  outcomes.  Experimental  data 
comes  from  single  or  few  unrepeatable  events.  Safety  restrictions,  lack  of  realism,  and 
unknown  composition  of  future  threats  make  AWE’s  “weakly  predictive”  at  best. 
Analysis  of  these  “weakly  predictive”  models  cannot  produce  quantitatively  significant 
rl?ita  Dewar’s  study  of  weakly  predictive  models  demonstrated  how  the  CU  framework 
could  be  used  to  adjudicate  clearly  defined  hypothesis.  Lucas  has  stated  that  significant 
up-front  analysis  is  required  to  maximize  experimental  results  from  an  AWE.  Decisions 
on  important  issues  require  an  objective  and  traceable  link  from  experimental  results  to 
decisions.  [Lucas,  Banks,  Vye,  1998] 

The  decision-to-experiment  ladder  is  clearly  founded  in  the  scientific  method. 
With  it,  experiments  are  designed  to  affect  decisions  and  those  decisions  are  based  on 
experiment  outcomes.  The  decision-to-experiment  ladder  includes:  issues,  decisions  to 
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be  made,  argument  to  support  the  decisions,  hypotheses  to  be  adjudicated,  experiments  to 
resolve  hypotheses,  and  analysis  and  measures  to  implement  decisions  [Lucas,  Banks, 
Vye,  1998] 


a.  Issues 

While  using  the  weakly  predictive  model  presented  in  AWEs,  the  primary 
issues  to  be  addressed  must  be  strictly  limited.  How  many  issues,  and  to  what  extent  they 
may  be  examined,  will  be  defended  by  the  successful  completion  of  the  decision-to- 
experiment  ladder  of  the  CU  framework.  [Lucas,  Banks,  Vye,  1998] 

b.  Decisions  to  be  Made 

The  CU  process  includes  specific  decisions  that  can  be  resolved  by  an 
analytical  process,  specifically  by  experimentation.  Potential  decisions  made  within  the 
decision  to  experiment  ladder  must  explicitly  stated.  Specificity  in  stating  issues  is 
critical  to  experimental  design.  Decisions  must  be  stated  precisely  enough  so  that 
experiments  can  be  properly  designed  to  test  them.  [Lucas,  Moore,  Vye,  1998] 

c.  Argument  to  Support  the  Decisions 

When  using  the  CU  framework,  users  must  identify  tentative  arguments 
that  are  needed  to  support  the  decisions  to  be  made  [Lucas,  Moore,  Vye,  1998].  By 
referencing  how  the  issues  relate  to  the  decisions  to  be  made,  analysts  know  exactly  the 
strength  of  the  argument  needed  to  make  the  decision.  Where  strong  prior  belief  exists  in 
the  decision  to  be  made,  very  little  experimental  evidence  may  be  required  to  make  the 
decision.  Where  substantial  evidence  is  required,  it  may  not  be  possible  to  resolve  the 
decision  given  the  limited  resources  of  the  experiment.  Arguments  must  be  “sufficient 
and  succinct.  Superfluous  arguments  lead  to  added  complexity  and  the  identification  of 
unmanageable  amounts  of  experiments"  [Lucas,  Moore,  Vye,  1998,  p8].  This 
identification  of  arguments  to  support  decisions  is  critical  in  the  design  of  the  experiment 
^d  is  much  more  productive  than  conducting  the  experiment  and  then  deducing  what  can 
be  inferred  from  the  evidence.  [Lucas,  Banks,  Vye,  1998] 
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d.  Hypotheses  to  be  Adjudicated 

In  the  CU  framework,  AWEs  must  be  designed  to  test  specific  hypotheses 
that  support  credible  arguments.  By  definition,  a  hypothesis  is  “an  assumption  used  as  a 
basis  for  an  argument  or  investigation  or  a  theory  that  explains  a  set  of  facts  and  can  be 
tested  by  further  investigation”  [Lucas,  Banks,  Vye,  1998,  pl8].  According  to  Lucas,  the 
hypotheses  used  in  the  TF  XXI  AWE  were  faulty  in  that  they  were  either  obviously  true 
or  were  not  specific  enough  to  be  subject  to  experimental  test.  [Lucas,  Banks,  Vye,  1998] 

e.  Experiments  to  Resolve  Hypotheses 

The  experiment  used  to  resolve  the  hypotheses  must  include  what  things 
will  be  varied  and  what  data  will  be  extracted.  The  experiments  in  the  AWE  process  can 
be  live,  virtual,  or  constructive,  as  required  to  satisfy  the  needs  of  the  hypothesis.  To 
maximize  the  analysis  capability  from  the  experiment,  significant  up-front  analysis  must 
be  completed.  Care  must  be  given  in  choosing  the  exact  variables  and  scenarios  for  the 
experiment.  It  might  be  determined  that  the  requirements  of  the  hypothesis  are  not 
feasible  within  the  given  constraints  of  the  exercise.  Experiments  should  be  as  objective 
as  possible.  Subjective  data  can  produce  erroneous  conclusions.  According  to  the  study 
by  Lucas,  the  Army  has  historically  been  effected  by  subtle  biases  that  have  prevented  its 
adaptation  to  new  technologies  [Lucas,  Moore,  Vye,  1998].  An  analysis  of  the 
experiment  and  its  assumptions  is  key  to  the  eventual  credible  use  of  the  resulting  data. 
[Lucas,  Banks,  Vye,  1998] 

f.  Analysis  and  Measures  to  Implement  Decisions 

For  the  experiment  to  be  used  credibly,  it  must  be  designed  to  support 
decisions.  Because  of  the  limited  data  gathered  from  AWEs,  there  must  be  an  “objective 
and  traceable  link  from  experimental  results  to  decisions  on  important  issues”  [Lucas, 
Banks,  Vye,  1998,  pl6].  The  procedures  and  products  of  the  experiment  must  be 
explicitly  stated  to  ensure  consistency,  traceability,  and  credibility.  [Lucas,  Banks,  Vye, 
1998] 
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D.  UTILIZATION  OF  EVALUATIONS 


The  Advanced  Warfighting  Experiments  are  evaluations  of  effectiveness  and 
efficiency  of  combat  operations.  No  matter  the  complexity  or  integrity  of  the  scientific 
process  of  the  exercise,  “in  the  end,  the  worth  of  evaluations  must  be  judged  by  their 
utility”  [Rossi,  1993].  To  achieve  maximum  benefit  from  experiments,  the  actual 
evaluation  must  be  tailored  to  the  specific  program,  the  stage  of  activity  in  the  program, 
and  the  needs  of  the  stakeholders  in  the  program.  The  evaluation  of  innovative  programs, 
such  as  the  digitization  of  the  Army,  may  require  leadership  to  rethink  some  aspects  of 
the  program,  including  the  program  objectives,  characteristics,  and  outcome  measures. 
[Rossi,  1993]. 

The  ability  of  an  organization  to  achieve  its  objectives  is  measured  by  three 
criteria:  (1)  the  extent  that  measurable  objectives  have  been  specified,  (2)  the  plausibility 
and  testability  of  assumptions  that  link  the  application  of  resources  to  the  program 
activities,  and  (3)  the  level  at  which  those  in  charge  of  the  program  have  the  motivation, 
ability,  and  authority  to  manage  and  implement  change.  Research  has  shown  that  the 
results  of  evaluations  might  not  be  used  to  refine  and  modify  programs  because 
leadership  may  resist,  be  uncooperative,  or  fail  to  grasp  the  purpose  of  the  studies.  Often, 
while  programs  may  have  a  number  of  objectives,  they  may  not  be  sufficiently  well 
defined  to  be  managed  to  desired  endstates,  may  not  have  valid  assumptions,  or  may  be 
invalidated  by  management’s  inability  to  implement  change.  [Rossi,  1993]. 

The  way  that  evaluations  are  used  can  be  classified  in  three  categories.  First, 
direct  or  instrumental  use  of  evaluations  means  that  the  specific  findings  and 
recommendations  are  directly  documented  and  implemented.  Second,  conceptual 
utilization  refers  to  the  general  use  of  evaluations  to  influence  thinking  about  issues. 
Conceptual  utilizations  do  not  lead  to  the  adoption  of  specific  programs  or  policies  but 
provide  evidence  that  supports  general  methods  and  approaches.  Finally,  persuasive 
utilization  applies  findings  to  support  or  refute  beliefs  or  to  attack  the  status  quo.  [Rossi, 
1993]. 

Five  factors  affect  the  utilization  of  evaluation  findings:  (1)  relevance  to  the 
problem,  (2)  communication  between  researchers  and  users,  (3)  information  processing 
by  users,  (4)  plausibility  of  research  results,  and  (5)  user  involvement  or  advocacy.  Also, 
leadership  will  be  influenced  by  the  perceived  truthfulness  of  the  evaluation. 
Management  will  measure  truthfulness  by  determining  if  the  research  was  conducted  by 
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proper  scientific  methods  and  if  the  results  are  compatible  with  their  experience, 
knowledge,  and  values.  [Rossi,  1993]. 

The  extent  that  the  data  obtained  from  the  TF  XXI  AWE  was  utilized  by  the  Army 
program  managers  involved  will  be  the  central  point  of  investigation  in  this  theses. 

E.  CHAPTER  CONCLUSIONS 

A  review  of  the  goals,  objectives,  and  limitations  of  the  TF  XXI  AWE  overlaid  on 
a  background  study  of  the  role  of  experiments  in  the  Army,  the  various  analyses  and 
recommendations  of  experimental  research  methods,  and  the  theoretical  utilization 
characteristics  of  evaluations,  raises  questions  about  the  possible  utility  of  the  information 
derived  from  the  TF  XXI AWTE  for  the  program  manager.  This  background  information 
will  be  considered  and  applied  to  the  results  of  the  objective  and  subjective  study  that 
follows. 

One  product  from  the  TF  XXI  AWE  was  a  performance  analysis  of  the 
participating  new  technology  initiatives.  OPTEC  provided  specific  recommendations  for 
each  initiative  in  its  Live  Experiment  Assessment  Report.  These  recommendations 
detailed  changes  to  user  requirements,  desired  technology  improvements,  and  integration 
issues.  This  thesis  studies  the  utility  of  the  AW^  to  program  managers,  as  discussed  by 
Rossi,  by  investigating  the  levels  at  which  these  recommendations  were  implemented  and 
the  factors  affecting  utilization  including  relevance  to  the  problem,  communication 
between  participants,  information  processing  by  users,  the  plausibility  of  research  results, 
and  user  involvement  [Rossi,  1993].  The  thesis  also  investigates  the  extent  to  which  the 
AWE  was  a  formative  evaluation  by  measuring  its  impact  on  the  programs  it  was  directed 
to  improve. 
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III.  EMPIRICAL  STUDY  DESCRIPTION 


A.  RESEARCH  AND  APPROACH 

1.  Research  Focus 

The  thesis  research  focused  on  gathering  ratings  from  program  managers  and  user 
representatives  as  to  the  degree  of  implementation  of  the  Operational  Test  and  Evaluation 
Command’s  specific  initiative  recommendations  relative  to  their  specific  program.  These 
recommendations  were  derived  from  the  Live  Experiment  Assessment  Report  for  the  TF 
XXI  AWE.  Secondly,  a  research  survey  was  conducted  on  the  perceived  validity  of  the 
recommendations  and  to  identify  contributing  factors  towards  a  program’s  ability  to 
benefit  from  participating  in  the  Advanced  Warfighting  Experiment. 

2.  Research  Approach 

Research  began  with  a  comprehensive  review  of  literature  addressing  the  goals, 
purpose,  techniques,  and  results  of  the  Army’s  experimental  process.  A  detailed  review 
of  the  TF  XXI  AWE  Experiment  Plan  and  OPTEC’s  Live  Experiment  Assessment  Report 
contributed  to  the  understanding  of  the  Army’s  experimental  program.  Background 
research  included  general  research  and  evaluation  methods  and  the  utilization  of 
evaluations.  Surveys  were  tailored  for  each  program  initiative  and  disseminated  to  each 
applicable  program  office  and  user  representative.  Research  included  follow-up 
interviews  of  pertinent  program  managers  and  users.  Analysis  was  then  conducted  to 
draw  conclusions  relevant  to  the  research  focus. 

B.  SAMPLING 


1.  Included  Initiatives 

This  research  does  not  include  an  analysis  of  the  Joint  Venture  XXI  doctrinal 
initiatives  or  organizational  change  initiatives  but  only  the  new  technology  developmental 
programs  participating  in  the  TF  XXI  AWE.  Of  the  ninety-two  emerging  technology 
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initiatives  included  in  the  experiment,  thirty-six  initiatives  were  included  in  the  research. 
Certain  initiatives  were  excluded  from  the  research  for  the  following  reasons;  (1)  the 
initiatives  was  not  evaluated  by  OPTEC  during  the  AWE,  (2)  OPTEC  did  not  provide  any 
substantive  recommendations  by  which  to  measure  implementation,  (3)  initiative  was  a 
doctrinal  or  organizational  change  with  no  material  program,  (4)  program  was 
subsequently  terminated  after  the  AWE  and  no  representatives  could  be  found  to  provide 
input  towards  the  research.  Of  the  thirty-six  initiatives  selected,  surveys  were  issued  to 
thirty-four.  One  initiative  was  subsequently  excluded  because  no  willing  participants 
could  be  found. 

2.  Selection  of  Respondents 

Seventy-two  respondents  were  identified  to  participate  in  the  survey.  First,  a 
representative  fi'om  the  responsible  program  management  office  was  identified  for  each 
of  the  thirty-six  initiatives.  Program  managers  were  selected  based  on  their  direct 
experience  with  the  AWE  and  their  ability  to  provide  programmatic  insights  into  the 
process  and  results  of  the  AWE.  Secondly,  a  user  representative  was  identified,  in  most 
cases  the  combat  developer  responsible  for  the  program.  Of  the  seventy-two  identified 
offices,  surveys  were  administered  to  sixty-seven. 

C.  SURVEY  DESIGN  AND  INSTRUMENT 

1.  Survey  Design 

The  survey  was  designed  to  collect  both  objective  and  subjective  data.  The 
objective  survey  items  were  designed  to  collect  data  on  the  specific  level  of 
implementation  of  recommendations  made  in  the  Live  Experiment  Assessment  Report. 
Subjective  items  were  included  to  gather  perceptions  and  opinions  firom  specific  program 
offices  and  user  representatives.  The  subjective  items  were  structured  from  the  RAND 
Corporation’s  application  of  the  “Credible  Uses”  (CU)  framework  to  Advanced 
Warfighting  Experiments  as  explained  in  detail  in  Chapter  n  of  this  thesis  [Lucas,  Banks, 
Vye,  1998]. 

The  CU  framework,  originally  documented  by  James  A.  Dewar  (et  al.)  [Dewar,  et 
al.],  is  based  on  a  decision-to-experiments  ladder  (DEL)  that  directly  links  experiments  to 
decisions  [Lucas,  Moore,  Vye].  Advanced  Warfighting  Experiments  are  limited  in  their 
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ability  to  predict  real  world  outcomes,  since  experimental  data  generally  comes  from 
single  or  few  unrepeatable  events.  Safety  restrictions,  lack  of  realism,  and  unknown 
composition  of  future  threats  make  AWE’s  “weakly  predictive”  at  best.  Analysis  of  these 
“weakly  predictive”  models  caimot  produce  quantitatively  significant  data.  However, 
Dewar’s  study  of  weakly  predictive  models  demonstrated  how  the  CU  framework  could 
be  used  to  adjudicate  clearly  defined  hypothesis  [Lucas,  Banks,  Vye].  The  items  in  the 
subjective  portion  of  the  survey  are  fi:amed  on  the  decision-to-experiment  ladder  to  allow 
participants  to  demonstrate  through  their  responses  the  extent  to  which  this  decision 
ladder  was  implemented  during  the  AWE. 

2.  Survey  Instrument 

The  survey  instrument  utilized  a  combination  of  ordinal  measurements  reflecting 
the  level  of  implementation  achieved  for  each  OPTEC  recommendation  and  factors 
affecting  implementation,  as  well  as  open-ended  questions  designed  to  gain  subjective 
perceptions  about  the  AWE  process.  The  survey  contained  foiu  sections. 

a.  Section  One 

Section  one  asked  respondents  to  rate  the  level  of  implementation  of  the 
specific  recommendations  made  for  their  program  initiative  by  the  Army  Operational  Test 
and  Evaluation  Command's  Live  Experiment  Assessment  Report.  Each  survey  was 
tailored  to  include  only  those  specific  recommendations  applicable  to  the  initiative  in 
question.  The  complete  list  of  specific  OPTEC  recommendations  is  included  as 
Appendix  A.  Participants  were  asked  to  indicate  the  term  that  best  described  the  extent 
that  the  recommendation  was  implemented;  FULLY,  MOSTLY,  LIMITED,  NOT  AT 
ALL.  Participants  were  also  asked  to  provide  narrative  comments  explaining  the  factors 
that  influenced  the  degree  of  implementation. 

b.  Section  Two 

Section  two  contained  questions  about  the  included  program's  experiences 
in  the  1997  TF  AWE.  Participants  were  asked  to  rate,  using  a  five-point  scale  (l=high, 
5=low),  questions  that  were  derived  from  the  decision-to-experiment  ladder  [Lucas, 
Moore,  Vye,  1998].  The  questions  covered  all  of  the  applicable  components  from  issue 
development  to  analysis  and  implementation.  Respondents  were  given  the  opportunity  to 
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give  a  brief  narrative  explanation  of  each  rating.  In  addition  to  the  objective  ratings, 
open-ended  questions  were  included  (see  questions  one,  six,  and  seven  in  Figure  2)  to 
gather  contributing  factors  and  characteristic  description  of  programs  best  positioned  to 
benefit  from  AWEs.  The  specific  questions  contained  in  Section  Two  are  listed  in  Figure 
2. 


Part  2: 

The  following  questions  gather  information  on  your  program’s  experience  in  the  1997  TF  XXI 
Advanced  Warfighting  Experiment.  In  the  area  given,  please  provide  your  perceptions  and 
opinions.  Where  applicable,  please  circle  the  appropriate  rating  that  best  answers  the  given 
question.  Please  provide  as  much  detailed  information  as  possible.  You  may  use  the  back  of 
this  form  or  attach  additional  pages. 


1.  What  were  the  specific  developmental  ISSUES  being  addressed  on  your  program  at  the  time 
of  its  participation  in  the  AWE  and  what  DECISIONS  were  to  be  made  from  gathered  data? 

Explain; 


2.  To  what  extent  were  you  able  to  tailor  or  influence  your  program’s  specific  activities  in 
the  AWE  to  relate  to  the  issues  and  decisions  you  faced  as  an  acquisition  manager? 

(High)  [1]  [2]  [3]  [4]  [5]  (Low) 

Explain; 

3.  How  valuable  were  the  data  and  recommendations  gained  from  participation  in  the  AWE 
in  making  decisions  as  an  acquisition  manager?  Explain. 

(High)  [1]  [2]  [3]  [4]  [5]  (Low) 

Explain; 

4.  To  what  extent  did  your  program  benefit  from  participating  in  the  AWE?  How? 

(High)  [1]  [2]  [3]  [4]  [5]  (Low) 

Explain; 

6.  What  were  the  contributing  factors  to  your  program’s  ability  to  benefit  from  participation  in 
the  AWE? 

Explain; 


7.  Based  on  your  program’s  experience  in  the  1 997  AWE,  describe  the  characteristics  of  a 
program  that  would  best  be  situated  to  benefit  from  participation  in  a  future  AWE. 

Explain; 

Figure  2.  Section  Two  of  Survey  Instixunent 
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Section  Three 


c. 


Section  three  gathered  respondent  opinions  on  reasons  why 
recommendations  made  by  the  1997  Advanced  Warfighting  Experiment  were  or  were  not 
fiilly  implemented.  The  questionnaire  attempted  to  include  an  exhaustive  list  of  potential 
impediments  or  positive  factors  for  implementation.  Participants  were  asked  to  rate  each 
of  the  items  on  a  scale  of  one  to  five.  A  rating  of  one  signified  a  highly  significant  factor 
on  the  degree  of  implementation.  A  rating  of  five  signified  that  an  item  was  NOT  an 
influencing  factor.  Participants  were  also  asked  to  place  a  mark  next  to  the  most  critical 
factor.  Section  three  of  the  survey  is  shown  as  Figure  3. 

Part  3: 

The  following  section  gathers  your  opinion  on  reasons  why  recommendations  made  by  the  1997 
Advanced  Warfighting  Experiment  were  or  were  not  fully  implemented.  Please  rate  each  of  the 
items  below  on  a  scale  of  1  to  5.  A  rating  of  1  signifies  a  highly  significant  factor  on  the  degree  of 
Implementation.  A  rating  of  5  signifies  that  an  Item  was  NOT  an  influencing  factor.  Please  place 
a  *  next  to  the  most  critical  factor.  The  first  set  of  items  are  possible  impediments  to 
implementation.  The  second  are  possible  positive  factors  for  implementation. 


Reasons  for  less  than  full  Implementation. 

Highly 

Significant 

“ 

Not 

Significant 

1 .  Lack  of  money  to  implement  recommendations  from  AWE. 

[1] 

[2] 

[3] 

[4] 

[5] 

2.  Lack  of  time  to  implement  recommendations  from  AWE. 

[1] 

[2] 

[3] 

[4] 

[5] 

3.  Test  data  and  recommendations  from  AWE  were  not  valid. 

[1] 

[2] 

[3] 

14] 

[5] 

4.  AWE  evaluators  did  not  understand  user’s  requirements. 

[1] 

[2] 

[3] 

[4] 

[5] 

5.  AWE  data  and  recommendations  were  not  accurate. 

[1] 

[2] 

[3] 

[4] 

[5] 

6.  AWE  recommendations  were  not  technically  feasible. 

[1] 

[2] 

[3] 

[4] 

[5] 

Factors  supporting  implementation. 

Highly 

Significant 

- 

Not 

Significant 

1 .  Issue  was  considered  high  priority  by  users. 

[1] 

[2] 

[3] 

[4] 

[5] 

2.  Issue  was  considered  high  priority  by  program  office. 

[1] 

[2] 

[3] 

[4] 

[5] 

3.  Issue  had  strong  political  backing. 

[1] 

[2] 

[3] 

[4] 

[5] 

4.  Recommended  actions  were  fully  funded. 

[1] 

[2] 

[3] 

[4] 

[5] 

Figure  3.  Section  Three  of  Survey  Instmment 
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d.  Section  Four 

In  section  four,  respondents  were  asked  to  provide  any  additional 
comments  on  their  program’s  participation  in  the  AWE  including  any  information  that 
might  assist  acquisition  managers  in  gaining  maximum  benefit  from  participation  in 
future  experiments  or  that  would  assist  planners  in  tailoring  future  experiments  to  better 
benefit  participating  programs. 

D.  DATA  COLLECTION 

First,  potential  respondents  from  the  selected  offices  were  contacted  by  email  to 
explain  the  survey  and  its  purpose.  The  thesis  topic  and  potential  derived  benefits  were 
explained.  Potential  respondents  were  asked  if  they  had  both  the  necessary  background 
experience  in  the  TF  XXI  AWE  and  the  estimated  time  of  one  hour  to  complete  the 
survey.  Surveys  were  then  administered  to  participants  by  issuance  and  submission  of 
Microsoft  Word  documents  via  electronic  mail. 

Interviews  were  conducted  in  person  with  seven  representatives  from  the  Army’s 
Night  Vision  Reconnaissance  Surveillance  and  Target  Acquisition  Office  at  Fort  Belvoir, 
Virginia  and  two  representatives  from  the  Program  Management  Office  for  Bradley 
Systems  at  the  Army's  Tank,  Automotive,  and  Armaments  Command  (TACOM)  in 
Detroit,  Michigan.  Interviews  were  conducted  face-to-face  during  a  combined  one  week 
on-site  visit  from  August  24  -  27,  1999.  These  interviews  were  designed  to  solicit  more 
detailed  information  addressing  the  same  questions  included  in  the  administered  survey. 
A  sample  of  questions  used  in  the  interviews  is  shown  in  Figure  4. 

All  respondents  to  the  written  survey  and  personal  interviews  were  informed  that 
the  research  would  be  conducted  on  a  non-attribution  basis  and  that  participants  would 
remain  anonymous.  It  was  believed  that  anonymous  input  would  result  in  a  more  candid 
disclosure  of  information. 

E.  LIMITATIONS  OF  RESEARCH  DESIGN 

The  following  is  a  description  of  the  major  factors  identified  prior  to  the  survey 
administration  that  could  limit  the  study. 
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1.  As  a  material  developer  and  acquisition  manager,  what  is  your  overall  perception  or 
impression  of  the  AWE  process? 

2.  What  kind  of  specific  feedback  did  you  receive  on  your  program’s  performance  in  the 
AWE?  Was  it  valid?  Could  you  take  advantage  and  utilize  the  feedback. 

3.  Has  your  program  undergone  any  major  changes  since  the  AWE? 

4.  How  did  the  maturity  of  your  program  effect  its  performance  at  the  AWE? 

5.  Were  you  able  to  tailor  the  analysis  plan  for  your  program  with  OPTEC? 

6.  How  did  your  program  benefit  from  participating  in  the  AWE? 

7.  What  were  the  risks  involved  with  participating  in  the  AWE?  How  were  those  risks 
mitigated? 

8.  What  amount  of  effort  was  involved  in  participation  in  terms  of  resources,  time, 
manpower,  or  dollars?  How  does  the  cost  of  participation  compare  to  the  benefits 
received? 

9.  What  role  did  politics  play  in  outcomes  and  perceptions  of  programs? 

10.  To  what  extent  was  the  AWE  a  marketing  effort  for  PMs? 

11.  How  would  you  characterize  a  program  that  would  stand  to  significantly  benefit  from 
participation? 


Figure  4.  Sample  Interview  Questions 


1.  Scope 

The  research  was  limited  to  programs  participating  in  the  TF  XXI  AWE  and  no 
other  Army  experiments.  The  available  programs  may  not  represent  the  full  breadth  of 
possible  outcomes  of  the  AWE  process.  Subsequent  Army  AWEs  may  have  been  refined 
to  invalidate  any  conclusions  resulting  from  this  study.  The  sample  of  included  programs 
might  not  be  representative  and  may  limit  the  relevance  and  validity  of  conclusions. 

Some  program  offices  have  been  disbanded  subsequent  to  the  TF  XXI  AWE. 
Representatives  for  certain  initiatives  were  not  available  to  contribute  to  the  study.  High 
levels  of  personnel  turnover  in  Army  agencies  might  lead  to  the  conclusion  that 
respondents  did  not  have  relevant  experience  to  evaluate  the  AWE.  In  some  cases,  details 
of  the  planning  process  and  results  of  the  AWE  were  not  docuniented  and  were  therefore 
unavailable  for  the  study. 
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2.  Reliability 


Reliability  is  “a  matter  of  whether  a  particular  technique,  applied  repeatedly  to  the 
same  object,  would  yield  the  same  result  each  time”  [Babbie,  1990,  pl32].  Although 
every  effort  was  made  to  conduct  the  research  in  a  rigorous  manner,  the  study  relied  on 
data  from  multiple  sources  with  unknown  biases.  All  data  were  based  on  opinions  and 
perceptions  that  were  not  independently  verified.  Depending  on  the  amount  of  data 
actually  collected  and  agreement  across  the  sample,  levels  of  confidence  can  be  measured 
to  qualify  conclusions. 

3.  Validity 

Validity  is  “the  extent  to  which  an  empirical  measure  adequately  reflects  the  real 
meaning  of  the  concept  under  consideration”  [Babbie,  1990,  pi 33].  The  inclusion  level 
participating  AWE  initiatives  studied  is  sufficient  to  create  valid  conclusions  oh  the 
outcomes  of  the  TF  XXI  AWE.  However,  any  derived  conclusions  might  not  be 
applicable  to  programs  participating  in  future  AWEs  due  to  changes  in  the  acquisition 
process  since  1997. 

F.  RESPONSE  STATISTICS 

Sixty-seven  surveys  were  administered  to  both  program  managers  and  user 
representatives  of  thirty-five  different  AWE  initiatives.  A  total  of  thirty-eight 
respondents  returned  completed  surveys  for  a  response  rate  of  56.7%.  Only  twenty-eight 
of  the  thirty-five  originally  included  initiatives  had  survey  responses.  The  remaining 
seven  initiatives  were  subsequently  excluded  from  the  thesis.  Of  the  twenty-eight 
included  initiatives,  only  ten  had  responses  from  both  the  program  manager  and  user 
representative.  While  the  cumulative  level  of  response  was  adequate  to  reach  general 
conclusions  on  the  acquisition  use  of  derived  AWE  data,  there  was  an  inadequate  level  of 
response  to  compare  program  manager  and  user  representative  input  or  to  measure  if  any 
significant  differences  existed  between  the  two  perspectives. 

Both  program  management  and  user  representative  offices  participated  in  the 
survey.  Completed  surveys  were  received  from  nineteen  program  management  offices 
and  nineteen  user  representative  offices.  The  program  management  office  surveys  were 
completed  by  program,  project,  or  product  managers.  The  users  were  represented  by  the 
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appropriate  Directorate  of  Combat  Developments  (DCD)  or  other  offices  to  include 
TRADOC  System  Managers  (TSMs)  and  Subject  Matter  Experts  (SMEs)  from  branch 
specific  doctrine  offices.  Respondents  included  military,  Department  of  Defense  (DoD) 
civilians,  and  support  contractors.  Military  respondents  ranged  from  Major  (0-4)  to 
Colonel  (0-6)  and  had  an  average  of  1.9  years  experience  on  the  program  in  question. 
Civilian  respondents  ranged  from  GS-12  to  GM-15  and  had  an  average  of  7.5  years 
experience  on  their  program.  The  minimum  amount  of  time  any  respondent  had  with 
their  program  was  12  months.  Also  included  were  four  civilian  contractors  who  directly 
supported  program  offices  or  user  agencies. 

Data  were  collected  on  the  maturity  of  programs  at  the  time  of  participation  in  the 
AWE.  Respondents  were  asked  to  indicate  their  "phase  of  development"  at  the  AWE. 
Responses  were  then  transcribed  into  five  categories  to  be  used  in  analysis.  Program 
maturity  is  summarized  in  Table  1. 


Table  1 .  Reported  Maturity  of  Programs  at  the  AWTE 


Matarity 

ilM 

Concept  Exploration  (CE)  or  Advanced  Technology 
Demonstration  (ATD) 

13 

Program  Definition  and  Risk  Reduction  (PDRR) 

4 

Engineering  and  Manufacturing  Development  (EMD) 

2 

Limited  Production  (LP)  or  Low  Rate  Initial 

Production  (LRIP) 

4 

Production,  Full  Rate  Production  (FRP),  or  Full  Scale 
Production  (FSP) 

5 

Overall,  given  the  scope,  reliability,  and  validity  of  the  response  demographics, 
the  data  are  adequately  representative  of  the  TF  XXI  AWTE  participating  program 
managers  and  user  representatives.  The  collected  data  are  sufficient  to  reach  findings  on 
the  use  of  AWEs  to  support  acquisition  decisions. 

G.  ANALYSIS  STRATEGY 

The  thesis  survey  asked  respondents  to  subjectively  rate  reconunendation 
implementation  and  perceptions  about  the  AWTE  using  a  number  scale.  For  this  analysis. 
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the  provided  numeric  ratings  were  inverted  (5=high;  l=low).  Arithmetic  means,  medians, 
and  modes  were  computed  for  each  question  to  determine  relationships  and  effects. 
Mean  is  defined  as  the  average  and  was  computed  by  dividing  the  sum  of  the  terms  by  the 
number  of  the  terms.  The  median  term  is  defined  as  that  value  of  an  ordered  set  below 
and  above  which  there  are  an  equal  number  of  values.  Mode  is  defined  as  the  most 
frequent  value  of  a  set  of  data.  A  confidence  interval  was  computed  that  should  include 
the  true  value  of  the  parameter  95%  of  the  time.  The  confidence  interval  was  computed 
as  plus  or  minus  95%  of  the  area  under  a  standard  normal  curve  multiplied  by  the  value  of 
the  standard  deviation  and  divided  by  the  square  root  of  the  sample  size.  For  most 
analysis,  medians  were  used  for  comparison  so  that  outliers  would  not  have  a  significant 
impact  on  results. 

Median  results  for  the  various  collected  factors  are  compared  against  one  another 
and  analyzed  to  determine  trends  and  overall  effects  on  the  ability  to  benefit  firom  the 
AWE.  The  Kruskal-Wallis  method  is  used  to  determine  a  P  value,  or  level  of  equality,  of 
the  various  factor  medians.  The  Kruskal-Wallis  method  is  computed  on  multiple  samples 
as  a  nonparametric  alternative  to  one-way  analysis  of  variance.  A  nonparametric  analysis 
implies  that  there  is  no  assumption  of  a  specific  distribution  for  the  population. 

The  Kruskal-Wallis  hypotheses  are;  HO:  the  population  medians  are  all  equal 
versus  HI:  the  medians  are  not  all  equal.  Kruskal-Wallis  H  is  calculated  on  the  basis  of 
sums  of  ranks  for  combined  groups.  Data  from  all  samples  are  ordered,  letting  n  be  the 
number  of  observations  in  any  particular  sample  and  N  is  the  number  of  observations  in 
all  samples  combined.  The  data  are  renumbered  fi-om  1  to  N,  with  1  corresponding  to  the 
lowest  score.  Using  these  rank  scores,  data  for  each  sample  are  listed  by  rank.  These 
rank  scores  are  added  up  for  each  sample,  and  the  sums  are  the  r  scores  in  the  formula 
below.  Also,  let  k  be  the  number  of  samples.  The  Kruskal-Wallis  H  is  computed  as:  H  = 
12/(N(N  -I-  1))*  SUM(r^/n)  -  3(N  +  1)  and  degrees  of  freedom  =  k  -  1.  H  is  distributed 
approximately  as  chi-square.  H  is  then  used  in  a  chi-square  table  with  (k  -  1)  degrees  of 
fireedom.  If  the  critical  value  of  chi-square  for  the  desired  significance  level  is  equal  to  or 
less  than  the  computed  H  value,  then  the  researcher  rejects  the  null  hypothesis  that  the 
samples  do  not  differ  on  the  criterion  variable.  In  the  analysis  that  follows,  the 
significance  level  for  each  Kruskal-Wallis  test  is  given.  [Garson,  1999] 

The  planned  method  of  analysis  included  a  review  of  the  respondent  surveys  to 
identify  the  following: 
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1.  The  average  level  of  implementation  of  all  included  initiative 
recommendations. 

2.  The  overall  perception  of  the  programs  ability  to  tailor  or  influence  the 
initiatives  specific  activities  in  the  AWE. 

3.  The  overall  perceived  value  of  the  data  and  recommendations  gained  from 
participation  in  the  AWE  as  acquisition  managers. 

4.  The  overall  perceived  benefit  gained  from  participating  in  the  AWE. 

5.  If  recommendations  reflected  trends  related  to 

-  level  of  program  maturity 

-  extent  to  which  program  was  able  to  tailor  or  influence  the  initiative’s 

activities  in  the  AWE 

-  perceived  value  of  the  data  and  recommendations  gained 

-  the  perceived  benefit  received  from  the  AWE 

6.  A  hierarchy  of  reasons  why  recommendations  made  were  or  were  not  fully 
implemented. 

Finally,  the  collected  subjective  comments  were  analyzed  to  draw  conclusions  to 
answer  the  primary  and  secondary  research  questions  and  develop  a  characteristic 
description  of  programs  that  are  best  positioned  to  gain  valued  investment  and 
requirements  information  from  participation  in  Advanced  Warfighting  Experiments. 
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IV.  FINDINGS  ON  THE  USE  OF  DERIVED  INFORMATION  IN  SUPPORTING 

ACQUISITION  DECISIONS 


A.  INTRODUCTION 

This  chapter  presents  the  summaries  of  the  subjective  ratings  gathered  from  the 
thesis  survey.  Section  A  of  this  chapter  discusses  the  statistical  measures  gathered  from 
the  survey  to  include: 

•  the  level  that  OPTEC  recommendations  were  implemented 

•  the  extent  to  which  managers  could  tailor  or  influence  their  program’s 
participation  in  the  AWE 

•  the  perceived  value  of  recommendations  gained  from  the  AWE 

•  the  level  that  programs  benefited  from  participation 

•  the  factors  influencing  implementation  or  lack  of  implementation 

Section  B  provides  an  analysis  of  recommendation  implementation  based  on  the 
above  program  issues.  Section  C  summarizes  this  chapter. 

B.  FINDINGS 

1.  Implementation  of  Recommendations 

Respondents  were  asked  to  indicate  the  term  that  best  described  the  extent  that 
specific  recommendations  made  by  the  OPTEC  Live  Experiment  Assessment  Report 
were  implemented  by  their  respective  programs.  The  survey  corresponding  with  each 
program  included  from  two  to  seven  recommendations  derived  from  the  Assessment 
Report.  The  total  number  of  recommendations  evaluated  was  one  hundred.  Possible 
responses  included  "Fully",  "Mostly",  "Limited",  or  "Not  at  All."  For  analysis  purposes, 
the  response  choices  were  given  an  ordinal  value  such  that  the  number  four  represented 
"Fully"  and  the  number  one  represented  "Not  at  All."  The  included  survey  instruments 
provided  one  hundred  total  recommendations  to  be  evaluated.  Where  differences  existed 
between  the  perceived  level  of  implementation  as  reported  by  the  program  manager  and 
that  reported  by  the  user  representative,  an  average  score  was  used. 
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The  average  level  of  reported  implementation  was  2.52,  translating  into  a  rating 
between  "Limited"  and  "Mostly".  The  median  response  was  3.0,  signifying  "Mostly" 
implemented.  A  histogram  of  reported  levels  of  implementation  is  shown  as  Figure  5. 
Of  all  the  included  recommendations,  52%  were  either  fiilly  or  mostly  implemented.  It  is 
important  to  note  that  there  was  a  fairly  even  split  between  those  recommendations  that 
were  fully  or  mostly  implemented  and  those  with  limited  or  no  implementation. 


Figure  5.  Reported  Levels  of  Recommendation  Implementation 

2.  AbUify  to  Tailor  or  Influence  a  Program’s  Activities  in  the  AWE 

Respondents  were  asked  to  rate  to  what  extent  they  were  able  to  tailor  or  influence 
their  program’s  specific  activities  in  the  AWE  to  relate  to  the  issues  and  decisions  they 
faced  as  acquisition  managers.  Examples  of  possible  tailoring  included  modifications  of 
scenarios  to  insure  data  availability  for  specific  issues,  influence  over  tactics  and 
techniques  used  by  participants  in  the  experiment,  and  contribution  to  the  data  collection 
or  analysis  plan.  For  the  purposes  of  analysis  in  this  thesis,  the  gathered  data  were 
converted  to  a  numerical  scale  from  one  to  five  with  five  being  a  high  ability  to  tailor  or 
influence  and  one  being  a  low  abihty.  A  total  of  twenty-eight  responses  were  evaluated. 
Ten  respondents  chose  not  to  answer  the  question. 

The  average,  level  of  ability  to  tailor  or  influence  activities  was  2.14  on  a  scale  of 
one  to  five,  translating  into  a  rating  slightly  lower  than  the  middle  of  the  range.  The 
median  response  was  1.0,  signifying  a  very  low  ability  to  tailor  their  activities.  The  mode 
of  responses  was  also  1.0.  A  histogram  of  reported  levels  of  implementation  is  shown  as 


Figure  6.  It  is  important  to  note  that  54%  of  all  respondents  reported  a  very  low  ability  to 
tailor  or  influence  activities. 


Figure  6.  Reported  Ability  to  Tailor  or  Influence  a  Program’s  Activities  in  the  AWE 

3.  Perceived  Value  of  Data  and  Recommendations  Gained  from 
Participation  in  AWE 

Respondents  were  asked  to  rate  the  value  of  the  data  and  recommendations  gained 
from  participating  in  the  AWE  in  making  decisions  as  acquisition  managers.  For  the 
ptuposes  of  analysis,  the  gathered  data  was  converted  to  a  numerical  scale  from  one  to 
five  with  five  signifying  that  the  data  was  of  high  value  and  one  signifying  low  value.  A 
total  of  thirty-one  responses  were  evaluated.  Seven  respondents  chose  not  to  answer  the 
question. 

The  average  perceived  value  of  the  data  and  recommendations  from  the  AWE  was 
3.10  on  a  scale  of  one  to  five,  translating  into  a  rating  directly  midpoint  of  the  range.  The 
median  response  was  also  3.0.  The  responses  had  no  significant  mode.  A  histogram  of 
reported  perceived  values  is  shown  as  Figure  7.  The  collected  data  indicate  that  42%  of 
the  included  programs  considered  the  value  of  the  data  and  recommendations  gained  as 
highly  valuable  (rated  4  or  5)  while  39%  perceived  the  value  as  very  low  (rated  1  or  2). 
The  degree  of  perceived  value  varied  significantly  among  initiatives. 
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Figure  7.  Reported  Value  of  Data  and  Recommendations  Gained  jfrom  AWE 

4.  Perceived  Beneflt  from  Participating  in  AWE 

Respondents  were  asked  to  rate  to  what  extent  their  program  benefited  from 
participating  in  the  AWE.  For  the  purposes  of  analysis,  the  gathered  data  were  converted 
to  a  numerical  scale  from  one  to  five  with  five  signifying  the  AWE  was  highly  beneficial 
and  one  signifying  that  it  was  less  beneficial.  A  total  of  thirty-four  responses  were 
evaluated.  Four  respondents  chose  not  to  answer  the  question. 

The  average  perceived  extent  of  benefiting  from  the  AWE  was  3.21  on  a  scale  of 
one  to  five,  translating  into  a  rating  slightly  above  the  midpoint  of  the  range.  The  median 
response  was  also  3.0.  The  responses  were  bimodal  at  3.0  and  5.0,  signifying  a  varying 
degree  of  benefiting.  A  histogram  of  reported  perceived  values  is  shown  as  Figure  8. 
The  data  indicates  that  45%  of  included  programs  benefited  from  the  AWE  (rated  4  or  5). 
However,  33%  of  programs  reported  little  to  no  benefit  (rated  1  or  2). 

5.  Factors  Affecting  the  Implementation  of  Recommendations 

Respondents  were  asked  to  rate  the  extent  that  specific  factors  supported  the 
implementation  of  AWE  recommendations  and  also  the  extent  that  specific  factors 
represented  reasons  why  recommendations  were  not  fully  implemented.  The  collected 
data  was  converted  to  a  numerical  scale  from  one  to  five  with  five  indicating  that  the 
factor  was  highly  significant  and  one  indicating  that  the  item  was  not  a  significant  factor. 
Six  items  were  presented  to  the  respondents  as  reasons  for  less  than  full  implementation. 
A  total  of  thirty-six  responses  were  evaluated  for  reasons  for  less  than  full 
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implementation.  Two  respondents  chose  not  to  rate  any  factors.  Four  items  were 
presented  to  the  respondents  as  factors  supporting  implementation.  A  total  of  thirty-four 
responses  supporting  implementation  were  evaluated.  Four  respondents  chose  not  to  rate 
any  factors  supporting  implementation. 


Figure  8.  Reported  Extent  that  Program  Benefited  from  Participation  in  the  AWE 

a.  Reasons  for  Less  Than  Full  Implementation 

As  shown  in  Figure  5,  seventy-five  percent  of  the  one  hundred  included 
recommendations  were  not  fully  implemented.  A  summary  of  the  medians  and  associated 
95%  confidence  intervals  for  the  factors  inhibiting  full  implementation  are  illustrated  in 
Figure  9.  According  to  the  survey,  a  lack  of  money  was  the  most  significant  reason  that 
recommendations  derived  from  participation  in  the  AWE  were  not  fiilly  implemented. 
This  factor  had  a  median  rating  of  4.0  with  a  rating  of  five  meaning  the  item  was  highly 
significant.  This  factor  rated  far  above  any  other  item,  including  the  second  most  cited 
reason,  that  test  data  and  recommendations  were  not  valid.  The  least  significant  item, 
according  to  respondents,  was  that  the  recommendations  were  not  technically  feasible. 
Other  items,  to  include  a  lack  of  time  to  implement,  AWE  evaluators  not  understanding 
the  user’s  requirements,  and  data  and  recommendations  not  being  accurate,  were  not 
reported  as  significant  factors. 
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Lack  of  Money  Lack  of  Time  Not  Accurate 

Not  Valid  Mteunderstood  Not  F=easjble 
Factor 

n  =  36 


Figure  9.  Median  Values  of  Reasons  for  Less  Than  Full  Implementation 


b.  Factors  Supporting  Implementation 


As  shown  in  Figure  5,  fifty-two  percent  of  the  one  hundred  included 
recommendations  were  either  fully  or  mostly  implemented.  The  most  significant  factor 
supporting  implementation  of  AWE  recommendations  was  that  the  recommendation  was 
considered  a  high  priority  by  the  user.  This  factor  had  a  median  value  of  4.0  on  a  scale 
from  one  to  five.  This  factor  was  higher  than  any  other  rated  item,  including  that  the 
issue  was  fully  funded,  was  considered  a  high  priority  by  the  program  office,  or  had  a 
strong  political  backing.  A  summary  of  the  factor  medians  and  associated  95% 
confidence  intervals  for  the  thirty-four  included  initiatives  are  graphically  illustrated  in 
Figure  10. 


42 


Figure  10.  Median  Values  of  Factors  Supporting  Implementation 


C.  ANALYSIS 


1.  The  Effects  of  Perceived  Value  of  Data  and  Recommendations  on 
Implementation 

The  perceived  value  of  AWE  data  and  recommendations  was  compared  to  the 
levels  of  recommendation  implementation.  Survey  responses  were  grouped  according  to 
reported  perceived  value  of  data.  Figure  1 1  illustrates  the  median  level  of 
recommendation  implementation  by  the  reported  value  of  data  and  the  associated  95% 
confidence  intervals.  Those  reporting  very  low  value  in  the  received  AWE  data  indicated 
a  median  implementation  level  of  2.0  on  a  scale  of  one  to  five  with  95%  confidence  that 
the  level  of  implementation  was  between  1.0  and  3.5.  Those  reporting  very  high  value  in 
the  received  A'WE  data  indicated  a  median  implementation  level  of  2.75  with  95% 
confidence  that  the  level  of  implementation  was  between  1.0  and  3.9.  Those  initiatives 
reporting  high  values  of  data  value  reported  slightly  higher  rates  of  recommendation 
implementation.  However,  the  wide  range  of  responses  and  resulting  large  confidence 
intervals  suggests  that  the  extent  that  recommendations  were  implemented  varied  only 
slightly  across  the  entire  reported  spectrum  of  perceived  value  of  data. 


43 


A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .757  showing  no  significant  difference  among  the 
median  ratings  of  the  value  of  AWE  data.  Therefore,  the  data  indicates  that  the  perceived 
value  of  AWE  data  and  recommendations  was  not  in  correlation  with  the  level  of 
recommendation  implementation. 


2  3  4  5- High 

Value  of  Data 
n  =  85 


Figure  11.  Median  Levels  of  Recommendation  Implementation  by  Value  of  Data 

To  determine  if  the  perceived  value  of  data  had  a  causative  effect  on  the  level  of 
recommendation  implementation,  a  comparison  was  made  of  the  reported  value  of  data 
and  implementation  levels.  Implementation  levels  were  grouped  into  two  samples:  (1) 
fully  and  mostly,  and  (2)  limited  and  not  at  all.  Figure  12  illustrates  the  median  level  of 
value  of  data  by  the  reported  implementation  level  and  the  associated  95%  confidence 
intervals.  Those  programs  with  recommendations  that  were  fully  or  mostly  implemented 
reported  a  median  value  of  data  equal  to  3.0.  Those  programs  with  recommendations  that 
had  limited  implementation,  or  no  implementation  at  all,  reported  a  median  value  of  data 
equal  to  2.0.  A  trend  is  evident  that  suggests  programs  with  high  levels  of 
recommendation  implementation  reported  higher  levels  of  value  of  data. 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .322  signifying  that  the  two  levels  of  implementation 
are  not  significantly  different  in  terms  of  the  value  of  the  data  received  from  the  AWE. 
Although  a  trend  is  evident  that  suggests  programs  with  high  levels  of  recommendation 
implementation  reported  higher  levels  of  value  of  data,  the  small  sample  size  and  large 
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variance  leads  to  a  statistically  insignificant  conclusion.  An  analysis  of  the  respondent 
comments  helps  to  explain  the  suggested  trend  and  variance  in  responses. 


Figure  12.  Median  Levels  of  Value  of  Data  by  Recommendation  Implementation 

A  program  manager  for  a  conceptually  new  communication  program  stated  that 
the  feedback  from  the  AWE  was  very  valuable  in  the  decision  making  process  because  it 
was  based  on  actual  user  feedback  and  very  specific  user  requirements.  However,  the 
recommendations  provided  by  the  AWE  Live  Assessment  Report  were  not  all  technically 
feasible.  The  program’s  technology  was  also  considered  commercial-of-the-shelf  (COTS) 
and  did  not  lend  itself  to  recommendation  implementation.  The  user  representative  from 
a  program  currently  in  full  production  stated  that  the  data  gained  was  highly  valuable  in 
developing  tactics,  techniques,  and  procedures.  However,  the  recommendations  requiring 
contractor  developmental  effort  could  not  be  implemented  because  of  a  lack  of  available 
research,  development,  and  procurement  funding  as  well  as  the  inability  to  be  included  in 
production  models  due  to  the  late  stage  of  maturity  for  the  initiative.  The  program 
manager  of  the  same  initiative  stated  that  the  value  of  the  data  was  only  moderately 
valuable  in  identifying  "connectivity  issues"  (the  interoperability  of  digital  systems) 
associated  with  TF  XXI  and  that  most  of  the  recommendations  fell  outside  of  the 
responsibility  of  the  program  management  office. 

Data  indicated  that  the  OPTEC  recommendations  were  either  fully  or  mostly 
implemented  by  one  program  in  the  concept  exploration  phase  of  development. 
However,  the  program  manager  of  this  initiative  stated  that  the  value  of  the  AWE  data 


45 


was  low  because  those  in  leadership  positions  were  not  willing  to  acknowledge  the 
demonstrated  capabilities  of  the  system.  Another  program  manager  stated  that  the 
recommendations  had  low  value  in  making  decisions  because  of  the  statistically 
insignificant  data  provided  by  an  unrepeatable  experiment. 

It  can  be  concluded  from  the  above  data  and  statements  that  the  perceived  value  of 
the  data  may  not  have  been  a  significant  factor  in  the  implementation  of  the  AWE 
recommendations.  The  trend  pattern  of  medians  suggests  a  positive  correlation. 
However,  the  small  sample  size  in  each  group  leads  to  a  lack  of  statistical  significance. 
Qualitative  comments  elaborated  the  rating  results  that  factors  other  than  perceived  value 
of  data  effected  the  recommendation  implementation.  Specifically,  participants 
considered  data  valuable  in  making  decisions  as  acquisition  managers  when  the 
recommendations  met  one  of  three  requirements.  First,  the  data  was  valuable  when  it 
provided  actual  user  feedback  on  specific  user  requirements.  Second,  data  was 
considered  valuable  when  it  contributed  to  the  development  of  tactics,  techniques,  and 
procedures.  Finally,  the  data  was  considered  valuable  when  it  was  provided  at  a  time 
when  it  could  be  instrumental  in  refining  requirements  and  design. 

2.  The  Effects  of  Perceived  Benefit  from  Participating  in  the  AWE  on 

Implementation 

The  perceived  benefit  from  participating  in  the  AWE  was  compared  to  the 
recommendation  implementation  levels.  An  acquisition  manager’s  perception  of  the 
extent  that  a  program  benefited  from  participating  in  the  AWE  was  reflected  in  the  level 
that  AWE  recommendations  were  implemented.  Survey  responses  were  grouped 
according  to  reported  perceived  benefit  gained  from  the  AWE.  Figure  13  illustrates  the 
median  level  of  recommendation  implementation  by  the  extent  that  a  program  benefited 
from  the  AWE  and  the  associated  95%  confidence  intervals.  The  programs  that  reported 
a  very  low  level  of  benefit  from  participating  in  the  AWE  indicated  a  median 
implementation  level  of  1.0  (not  at  all)  on  a  scale  of  one  to  five  with  a  95%  confidence 
that  the  implementation  was  between  1.0  and  2.0.  Those  reporting  a  very  high  level  of 
benefit  indicated  a  median  implementation  of  2.8  (mostly  implemented)  with  a 
confidence  interval  from  1.5  to  3.0. 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .036  showing  that  the  median  levels  of 
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implementation  were  significantly  different  when  comparing  levels  of  reported  perceived 
benefit.  Therefore,  the  data  indicates  that  the  perceived  benefit  of  the  AWE  was  in 
correlation  with  the  level  of  recommendation  implementation.  The  data  indicates  that 
those  programs  receiving  moderate  to  high  benefit  from  the  AWE  were  more  likely  to 
implement  the  OPTEC  recommendations  then  those  reporting  low  or  very  low. 


Figure  13.  Median  Levels  of  Recommendation  Implementation  by  Perceived  Benefit 

To  determine  if  the  level  of  recommendation  implementation  had  a  causative 
effect  on  the  perceived  benefit  from  AWE  participation,  a  comparison  was  made  of  the 
reported  perceived  benefit  and  implementation  levels.  Implementation  levels  were 
grouped  into  two  samples:  (1)  fully  and  mostly,  and  (2)  limited  and  not  at  all.  Figure  14 
illustrates  the  median  level  of  perceived  benefit  by  the  reported  implementation  level  and 
the  associated  95%  confidence  intervals.  Those  programs  with  recommendations  that 
were  fully  or  mostly  implemented  reported  a  median  perceived  benefit  equal  to  4.0. 
Those  programs  with  recommendations  that  had  limited  implementation,  or  no 
implementation  at  all,  reported  a  median  perceived  benefit  of  3.0.  A  trend  is  evident  that 
suggests  programs  with  high  levels  of  recommendation  implementation  reported  higher 
levels  perceived  benefit. 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .117  signifying  the  difference  in  the  two 
implementations  levels  are  not  significantly  different  in  terms  of  the  perceived  benefit  of 
the  AWE.  However,  this  difference  comes  close  to  the  critical  value  of  p  <  .  10  often  used 
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for  small  sample  tests.  The  small  sample  size  and  variance  leads  to  a  statistically 
inconclusive  finding.  However,  the  data  shows  that  their  is  some  effect.  An  analysis  of 
the  respondent  comments  helps  to  explain  the  suggested  trend  and  variance  in  responses. 


Figure  14.  Median  Levels  of  Perceived  Benefit  by  Reconmiendation  Implementation 

A  program  manager  directing  a  pre-milestone  n  initiative  stated  that  the  AWE 
provided  tremendous  insight  for  Program  Definition  and  Risk  Reduction  (PDRR).  All  of 
the  associated  AWE  reconunendations  for  this  initiative  were  fully  implemented.  It  is 
interesting  to  note  that  the  mean  implementation  level  in  those  programs  receiving  the 
most  benefit  firom  the  AWE  (high  =  5)  was  lower  than  those  reporting  moderate  of 
moderately-high  levels  of  benefit  (3-4).  The  mean  recommendation  implementation  level 
of  these  programs  receiving  high  benefit  was  reduced  by  two  specific  initiatives.  Both 
initiatives  reported  a  high  level  of  benefit  received  with  no  implementation  of 
reconunendations  (not  at  all  =  1).  The  first  program  stated  that  none  of  the 
recommendations  fit  any  of  the  user’s  requirements.  The  user  representative  for  the 
program  stated  that  the  high  level  of  benefit  was  attributed  to  the  attention  and  exposiue 
that  the  initiative’s  performance  generated  at  the  AWE.  The  second  program  was  a 
logistics  and  support  system  that,  according  to  the  program  manager,  was  not  adequately 
stressed  in  the  AWE  environment.  None  of  the  recommendations  were  implemented 
because  it  was  felt  that  the  data  and  reconunendations  were  not  valid  or  accurate.  The 
program  manager  stated  that  the  focus  of  the  AWE  was  on  combat  systems  and  not  on  the 


associated  support  systems.  The  main  benefit  this  program  manager  reported  was  critical 
feedback  from  the  user  on  required  functionality. 

The  high  level  of  implementation  for  those  programs  reporting  moderate  benefit 
(3)  from  the  AWE  was  effected  mostly  by  one  initiative  that  reported  full  implementation 
of  all  AWE  recommendations.  The  program  reported,  however,  that  all  but  one  of  the 
shortcomings  was  identified  prior  to  the  AWE  and  that  the  AWE  was  not  instrumental  in 
identifying  these  problems.  Therefore,  the  program  manager  reported  high  levels  of 
recommendation  implementation  but  a  moderate  benefit  from  the  AWE. 

It  can  be  shown  from  the  collected  data  and  comments  that  the  level  of 
recommendation  implementation  and  the  perceived  level  of  benefit  from  the  AWE  were 
generally  related.  It  cannot  be  concluded,  however,  that  die  implementation  of  AWE 
recommendations  was  solely  responsible  for  a  respondents  perception  of  benefit,  as 
illustrated  by  the  provided  comments  and  slightly  lower  recommendation  implementation 
levels  for  those  reporting  a  high  amount  of  perceived  benefit.  A  wide  range  of 
confounding  factors  effected  an  initiatives  ability  to  benefit  from  the  AWE.  These 
contributing  factors  to  a  program’s  ability  to  benefit  from  participation  in  the  AWE  are 
discussed  in  Chapter  V. 

3.  Effects  of  Tailoring  a  Program’s  Activities  in  the  AWE  on 
Implementation,  Value  of  Data,  and  Perceived  Benefit 

The  reported  ability  to  tailor  or  influence  a  program’s  activities  in  the  AWE  was 
compared  to  collected  data  on  recommendation  implementation.  Tailoring  was  defined 
as  the  ability  to  influence  the  program’s  activities  to  relate  to  the  issues  and  decisions 
facing  the  program.  Additionally,  comparisons  were  made  between  the  ability  to  tailor 
and  perceived  value  of  data  as  well  as  the  ability  to  tailor  and  the  reported  level  of 
benefiting  from  AWE  participation.  For  the  following  comparisons,  survey  responses 
were  grouped  according  to  reported  ability  to  tailor  or  influence  program  activities  in  the 
AWE. 


a.  Tailoring  and  Implementation 

Figure  15  illustrates  the  median  level  of  recommendation  implementation  by  the 
extent  that  a  program  was  able  to  tailor  and  the  associated  95%  confidence  intervals.  The 
median  implementation  level  reported  for  those  programs  with  a  low  ability  to  tailor  or 
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influence  was  2.0  (limited  implementation)  on  a  scale  of  one  to  five  with  a  95% 
confidence  that  the  value  was  between  1.0  and  3.0.  The  median  implementation  level 
reported  for  those  programs  with  a  high  ability  to  tailor  their  activities  was  3.5  (mostly  to 
fully  implemented)  with  an  associated  confidence  level  between  2.6  and  3.9. 
Implementation  levels  increased  as  levels  of  tailoring  increased  for  every  factor  but  one. . 
The  low  median  implementation  level  for  the  tailoring  factor  of  four  can  be  attributed  to  a 
significantly  low  sample  consisting  of  only  one  initiative.  This  initiative  was  a 
production  item  that  lacked  developmental  funds  to  implement  the  derived 
recommendations. 
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Figure  15.  Median  Levels  of  Recommendation  Implementation  by  Level  of  Tailoring 


A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .062.  Given  a  significance  criterion  of  p  <  .10,  this 
demonstrates  that  the  levels  of  tailoring  differ  significantly  in  terms  of  the  degree  of 
implementation  of  recommendations.  Therefore,  the  data  indicate  that  the  level  of 
recommendation  implementation  level  was  in  correlation  with  the  reported  ability  to 
tailor.  The  collected  data  suggest  that  as  a  program  manager’s  ability  to  tailor  activities  in 
the  AWE  to  relate  to  the  program’s  issues  and  decisions  increases,  so  does  the 
implementation  level  of  derived  recommendations 

To  determine  if  the  level  of  tailoring  had  a  causative  effect  on  the  level  of 
recommendation  implementation,  a  comparison  was  made  of  the  reported  ability  to  tailor 
and  recommendation  implementation  levels.  Implementation  levels  were  grouped  into 
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two  samples:  (1)  fully  and  mostly,  and  (2)  limited  and  not  at  all.  Figure  16  illustrates  the 
median  level  of  tailoring  by  the  reported  implementation  level  and  the  associated  95% 
confidence  intervals.  Those  programs  with  recommendations  that  were  fully  or  mostly 
implemented  reported  a  median  perceived  benefit  equal  to  2.0.  Those  programs  with 
recommendations  that  had  limited  implementation,  or  no  implementation  at  all,  reported  a 
median  perceived  benefit  of  1.0.  A  trend  is  evident  that  suggests  programs  with  high 
levels  of  recommendation  implementation  reported  higher  levels  ability  to  tailor. 


Figure  16.  Median  Levels  of  Tailoring  by  Recommendation  Implementation 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .091.  Given  a  significance  criterion  of  p  <  .10,  this 
analysis  shows  the  higher  implementation  group  reported  significantly  more  tailoring  than 
the  low  implementation  group. 

b.  Tailoring  and  Value  of  Data 

Figure  17  illustrates  the  median  level  of  perceived  data  value  by  the  extent  that  a 
program  was  able  to  tailor  and  the  associated  95%  confidence  intervals.  The  median 
value  of  data  and  recommendations  reported  for  those  programs  with  a  low  ability  to 
tailor  or  influence  was  3.0  on  a  scale  of  one  to  five.  The  median  value  of  data  reported 
for  those  programs  with  a  high  ability  to  tailor  their  activities  was  5.0.  The  very  low 
sample  size  prevented  the  calculation  of  confidence  intervals  for  some  levels  of  tailoring. 
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A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the  sample 
medians.  The  test  resulted  in  a  P  of  .042  demonstrates  that  the  levels  of  tailoring  differ 
significantly  in  terms  of  the  rated  value  of  data  resulting  from  the  AWE.  A  trend  is 
evident  in  the  data  that  suggests  as  a  program  manager’s  ability  to  tailor  activities  in  the 
AWE  increases,  so  increases  the  perceived  value  of  the  data  and  recommendations  in 
making  acquisition  management  decisions.  However,  the  wide  range  of  responses  and 
associated  confidence  intervals  requires  further  analysis  of  provided  comments  to  reach  a 
conclusion  on  the  correlation  of  tailoring  ability  with  value  of  data.  This  concept  is 
explored  further  in  Chapter  V. 

c.  Tailoring  and  Perceived  Benefit 

Figure  18  illustrates  the  median  level  of  perceived  benefit  by  the  extent  that  a 
program  was  able  to  tailor  activities  in  the  AWTE  and  the  associated  95%  confidence 
intervals.  The  median  perceived  benefit  reported  for  those  programs  with  a  low  ability  to 
tailor  or  influence  was  3.0  on  a  scale  of  one  to  five.  The  median  value  reported  for 
perceived  benefit  of  those  programs  with  a  high  ability  to  tailor  their  activities  was  5.0. 
The  very  low  sample  size  prevented  the  calculation  of  confidence  intervals  for  some 
levels  of  perceived  benefit. 
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Figure  18.  Median  Levels  Perceived  Benefit  by  Level  of  Tailoring  Ability 


A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the 
sample  medians.  The  test  resulted  in  a  P  of  .054  demonstrates  that  the  tailoring  levels 
differed  significantly  in  their  reported  benefits  of  the  AWE.  The  data  indicate  that  a  high 
ability  to  tailor  or  influence  activities  in  the  AWE  had  a  correlation  with  the  perceived 
benefit  gained.  Those  initiatives  reporting  a  low  ability  to  tailor,  generally  had  low  levels 
of  perceived  benefit. 

d.  Conclusions  on  Ability  to  Tailor 

It  can  be  concluded  that  the  extent  that  a  manager  was  able  to  tailor  or 
influence  a  program’s  specific  activities  in  the  AWE  to  relate  to  the  program’s  acquisition 
issues  and  decisions  directly  contributed  to  the  level  of  recommendation  implementation, 
perceived  value  of  data,  and  the  extent  that  the  program  benefited  from  participating. 
However,  the  wide  range  of  responses  and  associated  confidence  intervals  requires  further 
analysis  of  provided  comments  to  reach  a  conclusion  on  the  effects  of  increased  tailoring 
ability.  These  trends  are  explored  in  more  detail  in  Chapter  V. 
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4.  The  Effects  of  Program  Maturity  on  Implementation,  Value  of  Data, 
and  Perceived  Beneflt 

The  program’s  level  of  maturity  while  at  the  AWE  was  compared  to  the  data 
collected  on  recommendation  implementation.  Additionally,  comparisons  were  made 
between  maturity  and  perceived  value  of  the  data  and  recommendations,  and  also 
between  maturity  and  the  perceived  benefit  from  AWE  participation. 

Programs  were  categorized  into  maturity  groupings  based  on  the  standard 
Department  of  Defense  acquisition  management  process  as  detailed  in  DoD  Regulation 
5000.2-R.  Phase  0  is  Concept  Exploration  (CE)  and  typically  consists  of  competitive, 
parallel  short-term  concept  studies.  Programs  that  were  Advanced  Technology 
Demonstrations  (ATDs)  are  also  included  in  this  group.  Phase  I  is  Program  Definition 
and  Risk  Reduction  (PDRR).  In  PDRR,  programs  become  defined  as  one  or  more 
concepts,  design  approaches,  and/or  parallel  technologies.  Phase  n  is  Engineering 
Manufacturing  and  Development  (EMD).  In  EMD,  the  most  promising  design 
approaches  are  translated  into  stable,  producible,  supportable,  and  cost-effective  designs. 
Also  manufacturing  and  production  processes  are  developed  and  system  capabilities  are 
demonstrated  through  testing.  Programs  late  in  EMD  begin  Low  Rate  Initial  Production 
(LRIP).  The  initiatives  that  have  begun  LRIP  have  been  group  under  Limited  Product 
(LP).  Phase  m  is  production,  fielding,  deployment,  and  operational  support  and  have 
been  grouped  as  production  items  (PROD).  [DoD  5(K)0.2-R]. 

a.  Maturity  and  Implementation 

Figure  19  illustrates  the  median  levels  of  recommendation  implementation 
by  program  maturity  and  the  associated  95%  confidence  intervals  for  the  medians.  This 
comparison  was  conducted  using  multiple  recommendations  for  each  of  the  thirty-seven 
included  programs,  resulting  in  a  sample  size  of  one  hundred.  The  highest  level  of 
recommendation  implementation  was  seen  on  those  programs  in  Engineering  and 
Manufacturing  Development  (EMD)  with  a  median  implementation  level  of  4.0  (fully 
implemented)  on  a  scale  of  one  to  five  with  a  95%  confidence  interval  from  3.0  to  4.0. 
The  lowest  levels  of  implementation  were  on  either  end  of  the  spectrum,  with  Concept 
Exploration  (CE)  and  Advanced  Technology  Demonstration  (ATD)  initiatives  reporting  a 
median  of  2.5  and  full  rate  production  items  having  a  mean  of  2.0. 


54 


Figure  19.  Median  Levels  of  Implementation  by  Maturity 


A  Kruskal- Wallis  H  test  was  conducted  to  determine  the  equality  of  the 
sample  medians.  The  test  resulted  in  a  P  of  .021.  This  finding  shows  that  the  maturity 
levels  differed  significantly  in  terms  of  the  degree  of  implementation  of 
recommendations.  The  data  indicates  a  trend  in  which  programs  with  high  levels  of 
recommendation  implementation  were  most  likely  midpoint  in  programmatic 
development. 


b.  Maturity  and  Value  of  Data 

Figure  20  illustrates  the  median  level  of  perceived  value  of  data  and 
recommendations  by  initiative  maturity  and  95%  confidence  levels  for  each  median  value 
for  each  of  the  thirty-seven  included  programs.  The  highest  median  score  for  value  of 
data  was  reported  by  those  programs  in  the  PDRR  phase  of  development,  with  a  median 
of  5.0  on  a  scale  of  one  to  five  with  a  95%  confidence  that  the  value  was  between  1.0  to 
5.0.  Programs  in  HMD  had  the  second  highest  median  at  4.0.  Programs  in  Limited 
Production  (LP)  and  Production  placed  the  lowest  value  on  the  data  received,  rating  at  2.0 
and  3.0  respectively.  Programs  in  CE  also  scored  low,  with  a  median  of  3.5. 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the 
sample  medians.  The  test  resulted  in  a  P  of  .458  indicating  no  significant  difference  in 
the  maturity  levels  of  programs  and  their  respective  ratings  of  data  value.  Therefore,  the 
data  indicates  that  value  of  data  and  recommendations  was  not  in  significant  correlation 


55 


with  level  of  maturity.  However,  the  data  indicates  a  possible  trend  between 
programmatic  maturity  and  the  perceived  value  of  the  data  and  recommendations  gained 
from  the  AWE  in  making  decisions  as  acquisition  managers.  Those  programs  in  the  mid- 
developmental  phases  of  acquisition  were  more  likely  to  report  high  perceived  value  of 
data  and  recommendations  than  those  programs  early  or  late  in  development.  Further 
analysis  of  provided  comments  is  required  to  determine  the  correlation  between  these  two 
factors.  This  correlation  is  explored  further  in  Chapter  V. 


Figure  20.  Median  Perceived  Value  of  Data  by  Maturity 


c.  Maturity  and  Perceived  Benefit 

Figure  21  details  median  levels  of  reported  benefit  by  program  phase  of 
development  with  95%  confidence  intervals  on  the  medians.  The  highest  reported  benefit 
was  seen  by  those  programs  in  PDRR  and  EMD  with  a  reported  median  value  of  4.0  on  a 
scale  of  one  to  five  with  a  95%  confidence  that  the  value  was  between  2.0  and  5.0  for 
PDRR  and  between  3.0  and  5.0  for  EMD.  Initiatives  in  production  reported  similar 
perceived  benefits.  Initiatives  in  CE  and  limited  production  had  median  values  of  3.0. 

A  Kruskal-Wallis  H  test  was  conducted  to  determine  the  equality  of  the 
sample  medians.  The  test  resulted  in  a  P  of  .235  signifying  no  difference  in  the  median 
reported  perceived  benefit  by  levels  of  maturity.  Therefore,  the  data  indicates  that 
perceived  benefit  was  not  in  correlation  with  level  of  maturity.  However,  there  is  a  trend, 
not  supported  by  statistical  findings,  that  those  programs  in  or  near  the  mid- 
developmental  phases  of  acquisition  were  more  likely  to  report  higher  levels  of  benefit 
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then  those  programs  in  other  phases.  Of  additional  importance,  those  programs  late  in 
production  reported  high  levels  of  benefit.  A  full  analysis  of  the  perceived  benefits  of 
AWE  participation,  as  reported  by  acquisition  managers  is  included  in  Chapter  V  and 
fully  explains  these  trends 


Figure  21.  Median  Perceived  Benefit  by  Maturity 


d.  Conclusions  on  the  Effects  of  Maturity 


It  can  be  concluded  that  a  program’s  level  of  maturity  related  to  the  level  of 
recommendation  implementation.  A  trend  is  evident  that  more  maturity  did  not  directly 
transcribe  into  a  higher  ability  to  benefit,  but  rather  points  existed  on  the  maturity 
spectrum  where  a  program  was  best  positioned  to  benefit  from  the  experiment. 
Additionally,  a  similar  pattern  of  relationship  seams  to  exist  between  a  program’s  level  of 
maturity  and  the  perceived  value  of  data.  The  preceding  data  indicates  that  a  program 
reported  the  highest  level  of  implementation,  perceived  value  of  data,  and  benefit  if  it  was 
in  the  mid-development  phases  at  the  time  of  the  AWE.  Programs  early  in  development, 
or  into  production,  generally  reported  lower  values  for  these  items.  However,  initiatives 
in  production  also  reported  high  levels  of  benefit,  even  though  they  did  not  fully 
implement  the  AWE  recommendations.  This  suggests  that  factors  other  than 
recommendation  implementation  may  have  effected  a  program’s  ability  to  benefit. 
Maturity’s  effect  on  a  program’s  ability  to  benefit  from  participating  in  the  AWE,  as  well 
as  other  factors,  are  explored  in  Chapter  V. 
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D.  CHAPTER  SUMMARY 


This  chapter  presents  the  sununaries  of  the  subjective  ratings  gathered  from  the 
thesis  survey  and  the  findings  on  the  use  of  information  derived  from  the  AWE  in 
supporting  acquisition  decisions.  Gathered  subjective  comments  are  provided  to  help 
interpret  the  data.  Overall,  programs  reported  that  fifty-two  percent  of  the  OPTEC 
reconunendations  from  the  AWE  Live  Assessment  Report  were  either  fully  or  mostly 
implemented.  Thirty  percent  of  recommendations  were  not  implemented  at  all.  A 
combined  fifty-four  percent  of  initiatives  reported  a  low  ability  to  tailor  their  activities  in 
the  AWE  to  relate  to  issues  and  decisions  they  faced  as  acquisition  managers. 
Respondents  indicated  only  a  moderate  benefit  from  participating  in  the  AWE  and  that 
the  data  and  recommendations  received  were  only  somewhat  valuable.  The  most  cited 
reason  for  recommendations  not  being  implemented  was  a  lack  of  funding.  The 
recommendation  was  most  likely  to  be  implemented  if  it  was  a  high  priority  of  the  user. 

The  quality  and  amount  of  collected  data  enables  one  to  reach  several  conclusions 
about  a  program’s  participation  in  the  AWE.  First,  the  perceived  value  of  the  data  and 
recommendations  did  not  have  a  statistically  significant  role  in  determining  if 
recommendations  were  implemented.  However,  those  programs  that  had  reported  the 
most  benefit  from  the  AWE  generally  had  higher  levels  of  recommendation 
implementation.  Secondly,  the  data  suggest  that  the  ability  to  tailor  activities  in  the  AWE 
contributed  to  the  resulting  implementation  levels.  The  data  indicates  a  significant 
relationship  between  degree  of  tailoring  and  the  level  of  implementation,  the  value  of  the 
data,  and  the  reported  benefits  of  the  AWE.  A  trend  was  evident  that  a  program’s  phase 
of  development  also  was  a  factor  in  receiving  benefit  from  the  AWE  in  that  programs  in 
or  near  the  mid-developmental  phases  reported  higher  levels  of  recommendation 
implementation  and  value  of  data  than  did  programs  that  were  early  or  late  in  their 
developmental  lifecycle.  This  trend  is  illustrated  by  the  dramatic  inverted  "U"  shape  seen 
in  Figures  18,  20,  and  21.  This  trend  is  explored  further  in  Oiapter  V. 

The  acquisition  manager’s  definition  of  the  term  "benefit"  as  it  applies  to  AWE 
initiatives  is  instrumental  in  conducting  further  analysis  of  the  outcomes  of  the  AWE. 
The  next  chapter  provides  further  analysis  of  the  factors  contributing  to  a  program’s 
ability  to  benefit  from  the  AWE  based  on  this  data  as  well  as  collected  subjective 
comments. 
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V.  ANALYSIS  OF  CONTRIBUTING  FACTORS  TO  BENEFITING  FROM 
ADVANCED  WARFIGHTING  EXPERIMENTS 

A.  INTRODUCTION 

This  chapter  presents  analysis  and  findings  on  the  contributing  factors  to 
acquisition  managers  benefiting  from  Advanced  Warfighting  Experiments.  The  analysis 
is  based  on  survey  ratings  data  presented  in  Chapter  IV,  subjective  comments  provided  by 
the  open-ended  questions  of  the  survey,  and  opinions  collected  during  personal 
interviews.  The  themes  discussed  in  this  chapter  represent  the  opinions  of  a  significant 
number  of  respondents  and  interviewees.  Section  B  discusses  how  the  level  of  benefit 
derived  from  the  TF  XXI  AWE  for  acquisition  managers  was  limited  due  to  the  nature 
and  environment  of  the  experiment.  Section  C  defines  "benefit"  as  perceived  by  the 
majority  of  acquisition  managers  and  discusses  the  various  benefits  received  by  TF  XXI 
AWE  participants.  Section  D  discusses  the  implementation  of  the  Decision  to 
Experiment  Ladder  (DEL)  during  the  AWE  and  how  it  effected  a  program’s  ability  to 
benefit.  The  DEL  is  analyzed  in  terms  of  objective  documentation,  utilization  of  the 
AWE,  and  a  manager’s  ability  to  tailor  his  initiative’s  participation  in  the  AWE.  Section 
E  provides  findings  on  the  effects  of  program  maturity  on  the  ability  to  benefit.  Section  F 
presents  risk  considerations  and  a  tentative  risk  and  benefit  analysis  plan  for  acquisition 
managers  considering  future  AWE  participation.  A  chapter  summary  is  included  as 
Section  G. 

B.  LIMITATIONS  DUE  TO  THE  NATURE  OF  THE  EXPERIMENT 

According  to  the  experiment’s  directive,  the  Task  Force  XXI  Advanced 
Warfighting  Experiment  was  not  designed  to  provide  significant  benefits  to  the 
acquisition  manager.  The  AWE  process  was  designed  primarily  to  determine  if 
digitization  of  the  force  resulted  in  increases  in  lethality,  survivability,  and  tempo  [ECC, 
1996].  The  nature  of  the  AWE  experiment  limited  the  potential  derived  benefits  for  the 
program  manager.  The  acknowledged  limitations  in  the  design  of  the  AWE  experiment 
caused  by  low  repeatability,  incomplete  data  collection,  the  interaction  of  effects  of 
varying  initiatives,  and  small  sample  size,  precluded  any  summative  findings.  Formative 
uses  of  the  AWE  by  the  acquisition  team  were  also  limited  because,  according  to  one 
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acquisition  manager,  "large  scale  exercises  do  not  lend  themselves  to  gathering  data  that 
is  useful."  According  to  many  program  managers,  the  experiment’s  results  were 
influenced  by  politics.  Finally,  the  applied  results  of  the  AWE  made  many  managers 
question  the  utility  of  the  exercise. 

1.  Design  of  the  Experiment 

The  design  of  the  AWE,  its  large  scale,  and  the  attempt  to  combine  training  with 
assessment,  did  not  allow  a  controlled,  repeatable  experiment.  Data  collection  was 
limited  to  insure  that  it  was  unobtrusive  to  the  training  unit.  The  large  number  of 
interacting  initiatives  participating  in  the  AWE  led  to  confounded  results.  Finally,  the 
relatively  small  sample  of  focused  iterations  and  unrepeatability  of  the  experiment 
prevented  the  collection  of  statistically  meaningful  data.  A  program  manager  stated  that 
the  AWE  data  and  recommendations  had  low  value  because  they  were  statistically 
insignificant  and  thus,  invalid.  The  limitations  in  the  experiment’s  design  were  a  result  of 
the  large  scale  of  the  exercise. 

2.  Scale  of  Experiment 

The  large  scale  of  the  AWE  prohibited  the  detailed  evaluation  of  a  system’s 
technical  performance.  When  asked  to  provide  conunents  that  would  assist  planners  in 
designing  experiments  that  would  better  benefit  participating  acquisition  managers,  a 
respondent  stated  that  "large  exercises  such  as  TF  XXI  do  not  permit  the  evaluation  of  a 
system's  increase  or  decrease  in  combat  effectiveness."  Another  official  stated  that 
"analysis  from  an  AWE  perspective  does  not  usually  attempt  to  discriminate  what  in 
particular  within  a  program  contributed  to  an  overall  experimental  outcome."  The 
previous  respondent  went  on  to  state  that  "the  amount  of  equipment  required  for  adequate 
data  collection  and  storage  is  cost  [prohibitive]... In  the  best  cases,  50%  of  the  actions 
performed  by  the  crews  is  captured.  This  causes  inaccurate  conclusions  in  most  cases." 
Another  respondent  stated: 
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These  large  scale  exercises,  that  have  multiple  new  systems  installed  on 
individual  vehicles,  negate  any  positive  outcome.  The  soldiers  are 
required  to  learn  many  new  systems  and  TTPs  [tactics,  techniques,  and 
procedures]  at  one  time  and  this  leads  to  mass  confusion.  In  many  cases 
the  systems  under  evaluation  were  simply  turned  off  or  never  turned  on. 

These  large  scale  exercises  are  a  political  asset  and  provide  no  concrete 
data  for  decision  makers  to  make  a  decision  on  the  system. 

3.  Political  Influence 

The  influence  of  politics  was  a  topic  discussed  by  many  acquisition  managers 
participating  in  the  survey.  While  "strong  political  backing"  was  the  lowest  rated  reason 
that  recommendations  were  implemented,  participants  believed  that  the  preconceived 
biases  of  senior  army  officials  effected  the  final  perceptions  of  a  program’s  performance 
at  the  AWE.  One  participant  stated,  "watch  the  politics.  On  day  one  of  the  AWE,  the 
event  was  declared  a  major  success.  Contrary  opinions  were  downgraded.  This  mindset 
started  with  the  then  TRADOC  Commander.  Ethics  and  testing  are  a  key  consideration." 
Another  respondent  stated  that  "incoming  biases  prevented  any  real  analysis  of  the 
system."  The  Army’s  Applique  program,  now  called  Force  XXI  Battle  Conunand  Brigade 
and  Below  (FBCB2)  was  often  cited  as  a  program  that  performed  marginally  in  the  AWE 
but  received  the  majority  of  post-AWE  funding  because  "Army  leadership  backed  them" 
and  the  system  was  "pushed  from  the  top". 

4.  Utility  of  Experiment 

Some  AWE  initiatives  received  high  marks  at  the  experiment  but  were 
subsequently  terminated,  leaving  the  acquisition  managers  of  those  programs  with 
negative  impressions  about  the  utility  of  the  experiment.  A  program  manager  for  an 
initiative  with  a  large  role  in  the  AWE  conmiented  that  their  system  had  great 
performance  but  Army  leadership  did  little  to  exploit  the  system.  A  user  representative 
for  a  now  terminated  program  stated: 

The  use  of  [the  initiative]  in  the  AWE  makes  me  question  the  criteria  for  a 
program’s  acceptance  into  the  AWE.  If  the  [initiative]  was  never  going  to 
be  accepted  it  should  not  have  been  included.. .Bottom  line  to  us  is  that 
even  though  this  [initiative]  proved  very  successful,  its  acceptance  was 
doomed  from  the  start.  In  the  end  a  great  deal  of  time,  money,  and  energy 
was  wasted  in  this  effort. 
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For  a  small  scale  initiative  participating  in  the  experiment,  the  AWE  was 
described  as  a  "lose/lose"  situation.  The  program  manager  stated  that  "there  were  only  a 
handful  of  high  visibility  programs.  While  the  AWE  was  a  great  way  of  proving  out  a 
system,  the  smaller  initiatives  couldn’t  get  any  visibility."  In  a  personal  interview,  a 
product  manager  summarized  that  the  AWE  was  a  "diversion."  It  was  "not  focused  on 
the  end  goal  of  getting  the  product  in  the  hands  of  the  soldiers.  The  AWE  took  away 
from  program  resources  to  play  in  a  sandbox." 

5.  Conclusions  on  Limitations 

Generally,  AWEs  have  not  been  designed  to  provide  significant  benefits  to  the 
acquisition  managers.  Managers  charged  with  the  development,  production,  and  fielding 
of  programs  participating  as  AWE  initiatives  must  take  active  measures  to  increase  the 
likelihood  of  achieving  benefits  for  the  program. 

C.  BENEFITS  TO  THE  ACQUISITION  MANAGER 

The  previous  section  reinforces  the  fact  that  the  AWE  was  not  designed  to 
purposefully  provide  data  to  acquisition  managers.  Nevertheless,  survey  participants 
identified  a  number  of  potential  benefits  from  participation.  The  extent  that  a  program 
benefited  from  participation  in  the  AWE  varied  widely.  Chapter  IV  showed  that  the 
perceived  benefit  from  the  AWE  was  only  moderate,  with  differing  opinions  from  various 
initiatives.  Subjective  comments  show  a  wide  interpretation  of  the  term  "benefit"  as  it 
applies  to  the  acquisition  manager.  One  respondent  stated  that  "the  feedback  obtained 
from  the  AWTE  was  very  valuable  to  the  decision  making  process.  It  is. ..easier  to 
structure  a  program  based  on  actual  user  feedback  and  very  specific  user  requirements 
obtained  during  the  AWE."  From  another  perspective,  a  respondent  stated  that  "there  was 
no  benefit  to  the  program  from  the  AWE.  It  cost  a  lot  of  money  to  provide  system 
support...we  feel  that  the  data.. .will  not  provide  any  additional  concrete  information  that 
can  be  used." 

A  significant  number  of  survey  participants  agreed  that  managers  must  determine 
if  potential  return  from  participating  in  an  AWE  outweigh  the  costs.  A  cost  analysis  of 
AWE  would  include  expenses  for  prototyping,  manning,  fielding,  schedule, 
transportation,  and  other  risk  categories.  One  program  manager  stated: 
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A  study  should  be  conducted  to  define  the  problems  to  be  resolved  and/or 
the  goals  of  the  AWE.  It  should  include  a  section  on  the  perceived 
benefits  to  each  participant.  Each  PM  whose  participation  is  required 
should  be  given  the  opportunity  to  provide  an  impact  statement  to  his 
program  if  he  participates.  He  should  propose  alternatives  to  minimize  the 
impact  to  his  current  efforts  that  have  funds  obligated. 

The  benefits  from  the  AWE  to  the  acquisition  manager  can  be  summarized  into 
the  following  categories:  (1)  marketing  and  exposure  of  program,  (2)  early  user  feedback, 
(3)  refinement  of  user  requirements,  (4)  development  of  tactics,  techniques,  and 
procedures,  (5)  follow-on  support  for  funding  and  production  decisions,  (6)  information 
on  integration,  interfaces,  and  interoperability,  and  (7)  exposure  of  developers  to  the 
user’s  environment. 

1.  Marketing  and  Exposure  of  Program 

Participating  acquisition  managers  were  in  agreement  that  no  PM  would 
participate  in  the  AWE  solely  for  the  marketing  effects.  However,  in  terms  of  potential 
AWE  benefits,  the  desire  for  positive  exposure  of  their  program  was  discussed  as  a 
benefit  by  respondents  more  than  any  other  aspect. 

The  importance  of  being  perceived  as  a  "member  of  the  team"  was  emphasized  by 
several  respondents.  One  program  manager  was  adamant  that  he  did  not  want  to 
participate  in  the  AWE  but  was  forced  into  participation  by  the  user  representative.  He 
stated  that  the  user  viewed  the  initiative's  participation  as  "a  marketing  opportunity." 
Another  program  claimed  that  they  participated  in  the  AWE  "to  increase  visibility  of  [the 
initiative's]  performance  capabilities  rather  than  to  address  developmental  issues."  A 
program  that  reported  a  veiy  high  level  of  benefit  from  the  AWE  stated  that  benefit  was 
"extremely  high  from  a  marketing  perspective.  The  [initiative]  demonstrated  to  the  world 
its  ability  to  dominate  the  battlefield." 

Marketing  was  most  often  described  as  an  ancillary  benefit.  In  a  personal 
interview,  a  senior  engineer  stated  that  the  AWE  process  is  a  good  way  to  expose  the 
program  but  was  not  just  marketing.  Participation  provided  exposure  of  the  program 
among  soldiers  and  Army  leadership.  Another  participant  said  in  an  interview,  that  the 
positive  response  to  his  program  benefited  him.  He  added,  however,  given  the  nature  of 
the  experiment,  the  results  and  data  could  not  be  valid. 
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2.  Early  User  Feedback 

Most  acquisition  programs  do  not  experience  any  hands-on  exposure  to  actual 
users  until  Initial  Operational  Test  and  Evaluation  (lOTE).  A  program  manager  stated 
that  any  time  you  can  get  your  product  into  the  hands  of  the  actual  user  early,  it  serves  to 
benefit  the  program.  lOTE  was  also  characterized  by  a  significant  number  of  respondents 
as  "too  structured"  to  provide  any  significant  feedback  from  the  user.  Feedback  during 
lOTE  is  also  "too  late  to  impact  system  design." 

It  is  still  unclear  if  benefits  associated  with  the  transference  of  AWE  data  to 
reduce  the  operational  test  and  evaluation  process  will  be  realized.  In  February  1997,  an 
article  discussing  this  subject  stated: 

WHiether  or  not  the  results  from  TF  XXI  will  contribute  to  the  formal  test 
and  evaluation... [process]. ..is  a  question  under  debate  right  now...The 
acquisition  community  has  averred  that  the  experiment  data  will  not 
receive  formal  consideration,  but  at  the  same  time  we’re  hearing 
comments  that  the  testing  community  needs  to  be  cognizant  of  the 
participation  in  TF  XXI  and  take  advantage  of  the  test  results  coming  out 
of  that.  [Lum,  1997,  p26] 

3.  Refinement  of  User  Requirements 

For  the  majority  of  respondents,  participation  in  the  AWE  provided  significant 
data  to  be  used  for  the  refinement  of  user  requirements.  Feedback  provided  from 
representative  users  experimenting  with  the  new  systems  in  a  combat  simulated 
environment  was  determined  to  be  of  high  value  to  many  acquisition  managers. 

For  one  program,  the  AWE  caused  a  complete  reanalysis  of  requirements  and  led 
to  the  development  of  new  key  performance  parameters.  One  manager  stated  that  "the 
AWTE  provides  a  good  forum  to  ascertain  whether  or  not  certain  requirements  are  feasible 
or  even  desirable  from  a  user  perspective."  A  user  representative  sunomarized: 

The  AWE  demonstrated  a  great  deal  of  utility  in  the  requirements 
development  process.  The  Army's  experimental  process  determined  what 
concepts  and  technologies  worked  and  how  they  can  be  made  to  work 
better.  The  process  also  identified  concepts  that  did  not  work  well. 

A  program  manager  directing  a  pre-milestone  II  initiative  stated  that  the  AWHE 
provided  tremendous  insight  for  program  definition  and  risk  reduction.  All  of  the 
associated  AWE  recommendations  for  this  initiative  were  fully  implemented.  A  user 
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representative  of  a  program  highly  concerned  with  refinement  of  requirements  stated  that 
"the  lessons  learned  from  actual  field  user  are  invaluable." 

Refinement  of  requirements  was  even  beneficial  to  some  programs  already  in 
production.  One  particular  initiative  utilizing  extensive  commercial-of-the  shelf  and  non- 
developmental  technology  had  received  a  Milestone  III  decision  and  moved  on  to 
production  without  any  operational  testing.  Participation  in  the  AWE  resulted  in  "major 
changes  in  system  architecture".  According  to  the  program  manager,  the  experiences 
gained  would  be  "utilized  in  the  design  of  follow-on  systems." 

Regardless  of  an  initiative’s  level  of  maturity,  most  acquisition  managers  were 
able  to  collect  valued  data  that  would  assist  in  the  refinement  of  system  requirements,  and 
contribute  to  the  development  of  a  more  effective  system. 

4.  Development  of  Tactics,  Techniques,  and  Procedures  (TTPs) 

The  development  of  tactics,  techniques,  and  procedures  (TTP)  was  identified  by  a 
significant  number  of  respondents  as  a  benefit  received  from  the  AWE.  TTP 
development  is  not  normally  a  function  of  the  acquisition  manager,  but  rather  a  function 
of  the  branch  specific  office  responsible  for  organizational  doctrine.  However,  the 
development  of  TTP  is  a  vital  step  in  the  process  of  gaining  user  support  for  a  technology. 
The  program  manager  for  one  AWE  system  described  his  initiative  as  a  whole  new 
concept  for  the  Army.  The  AWE  helped  to  refine  tactics,  techniques,  and  procedures  in 
an  area  that  previously  had  no  doctrine.  Another  manager  stated  that  the  AWE  confirmed 
"a  concept  in  an  operational  environment  without  a  commitment  to  expensive,  pre- 
production  hardware." 

5.  Follow-on  Support  for  Funding  and  Production  Decisions 

The  nature  of  today’s  defense  acquisition  environment  includes  a  constant  struggle 
for  program  funding.  A  former  acquisition  manager  stated  that  he  spent  twenty-five 
percent  of  his  time  as  a  PM  dealing  with  funding  issues  on  his  program.  Many 
respondents  stated  that  one  of  their  goals  in  AWE  participation  was  to  gain  data  to 
support  funding  requests  and  to  "get  a  foot  in  the  door  for  production  funding". 

The  Army’s  Warfighting  Rapid  Acquisition  Program  (WRAP)  was  tied  to  the  TF 
XXI  AWE.  The  WRAP  process  was  designed  to  accelerate  the  fielding  of  advanced 
systems  and  technologies  from  AWEs,  Advanced  Technology  Demonstrations  (ATDs) 
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and  Advanced  Concept  Technology  Demonstrations  (ACTDs).  WRAP  links  TRADOC’s 
battle  labs  to  the  formal  acquisition  process  so  that  compelling  successes  quickly 
transition  from  experimentation  to  enhanced  warfighting  capability.  The  TRADOC 
Commander  nominates  candidates  for  WRAP  after  a  review  based  on  urgency  of  need 
and  compelling  success.  Approved  candidates  are  then  ranked  by  priority  and  given  two- 
year  funding  for  operational  prototypes.  [Monnett  and  Rheinlander,  1996] 

Several  AWE  initiatives  desired  publicity  to  secure  WRAP  funding  and  used 
AWE  results  as  a  demonstration  of  compelling  success.  A  program  that  was  ultimately 
selected  for  WRAP  commented  that  "the  data  and  recommendations  gained  from  the 
AWE  were  critical  to  the  system’s  success  as  a  WRAP  candidate.  Without  the  WRAP 
funding,  the  program  was  an  unfunded  requirement  with  little  probability  of  being 
funded." 

The  attainment  of  production  funding  was  of  particular  concern  for  several 
initiatives  with  no  programmed  dollars  beyond  the  AWE.  The  Department  of  the  Army 
specifically  delayed  production  decisions  on  some  programs  until  the  AWE  was 
completed.  During  a  personal  interview,  a  PM  stated  that  Army  leadership  "expected 
great  things  of  the  AWE."  The  manager  of  a  now  terminated  program  commented  "if  the 
program  could  have  shown  merit  at  the  AWE  as  a  last  chance  effort,  then  it  would  have 
been  a  good  investment.  We  had  nothing  to  lose." 

The  data  show  that  initiatives  facing  future  funding  decisions  stand  to  receive 
significant  benefit  from  AWEs  if  compelling  success  can  be  demonstrated  in  an  AWE. 

6.  Information  on  Integration,  Interfaces,  and  Interoperability 

Integrating,  interfacing,  and  interoperability  of  new  technologies  within  the 
current  Army  force  sfructure  is  a  constant  challenge  among  acquisition  programs.  AWE 
participants  collected  data  to  meet  future  integration,  interface,  and  interoperability 
requirements.  This  data  provided  significant  benefit  to  acquisition  managers. 

Integration  in  acquisition  is  defined  as  the  incorporation  of  a  system  as  an  equal 
into  an  organization.  A  system  interface  is  a  common  boimdary  between  two  associated 
systems  across  which  information  may  flow.  Literoperability  is  defined  as  the  ability  of  a 
system  to  provide  services  to  and  accept  services  from  other  systems  to  enable  them  to 
operate  effectively  together.  [CALL,  1999]. 
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One  program  manager  explained  how  AWE  participation  benefited  the  program  in 
terms  of  integration,  interfaces,  and  interoperability: 

ffistorically  each  PM  had  his  own  requirements  to  implement  and  did  not 
feel  justified  in  experimenting  with  interoperability.  Each  PM  would 
develop  an  interface  to  other  systems  using  the  minimum  resources  to  get 
the  job  done.  The  interface  was  implemented  from  a  paper  design  based 
on  the  data  that  needed  to  be  shared.  In  the  AWE  each  [initiative]  got  the 
opportunity  to  view  the  operational  picture  as  presented  by  each  of  the 
other  [initiatives]. 

According  to  program  managers,  operational  testing  does  not  provide  the  required 
data  to  facilitate  integration,  interfacing,  and  interoperability.  The  AWE,  however, 
provides  much  needed  information  on  how  a  program  "interfaces  with  the  force." 
Reportedly,  the  AWE  was  a  "great  way  of  proving  out  hardware"  and  a  "great  way  to  test 
integration."  A  program  manager  stated  that  it  is  "always  worthwhile  to  exercise 
interoperability  with  other  systems  to  identify  nuances  and  characteristics  of  other 
systems  that  must  be  ’designed  to’." 

7.  Exposure  of  Developers  to  the  User’s  Environment 

By  participating  in  the  AWE,  acquisition  managers  and  program  engineers 
received  exposure  to  the  user’s  environment  in  a  realistic  combat  simulation.  Most  of  the 
participating  managers  lacked  any  real  experience  in  military  field  operations.  This 
exposure  was  cited  as  a  significant  benefit  from  AWE  participation. 

During  the  AWE,  program  managers  were  allowed  to  observe  the  experiment 
firsthand.  A  participating  program  manager  stated  that  the  AWE  was  beneficial  because 
it  "allowed  for  realtime  feedback  to  answer  questions  and  understand  deployment  issues." 
A  senior  official  with  oversight  of  multiple  initiatives  in  the  AWE  stated  during  an 
interview  that  the  AWE  had  "additional  intangible  benefits"  to  the  program  manager, 
beyond  the  benefits  derived  from  official  collected  data.  The  AWE  provided  acquisition 
managers  "early  exposure  to  the  customer."  According  to  the  official,  most  of  his 
engineers  and  program  managers  have  never  been  on  active  duty.  The  AWE  provided  an 
opportunity  for  these  personnel  to  experience  the  user's  environment  firsthand.  The  AWE 
also  allowed  the  engineers  and  program  managers  to  gather  data  on  peripheral  programs 
and  to  learn  the  state  of  technology  across  the  force.  Another  program  manager  stated 
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that  "the  biggest  benefit  was  firom  debriefing  the  soldiers  that  used  the  equipment.  They 
provided  more  information,  all  be  it  subjective,  than  any  questionnaire  did." 

The  data  clearly  show  that  acquisition  managers  who  lack  significant  experience 
in  military  field  operations  are  positioned  to  receive  significant  benefit  from  the  AWE  by 
gaining  an  understanding  of  the  environment  and  scenarios  in  which  their  respective 
systems  will  operate. 

D.  THE  DECISION  TO  EXPERIMENT  LADDER  (DEL) 

The  decision-to-experiments  ladder  (DEL)  discussed  in  Chapter  II,  directly  links 
experiments  to  decisions  and  is  designed  .to  increase  the  utility  of  experiments.  The  DEL 
includes:  issues,  decisions  to  be  made,  argument  to  support  the  decisions,  hypotheses  to 
be  adjudicated,  experiments  to  resolve  hypotheses,  and  analysis  and  measures  to 
implement  decisions.  [Lucas,  Banks,  Vye,  1998] 

The  level  that  the  DEL  was  implemented  in  the  AWE  appeared  to  directly  effect 
an  initiative’s  ability  to  benefit  from  the  experiment.  The  analysis  of  survey  responses 
provides  evidence  that  those  program  managers  who  could  provide  a  detailed  explanation 
of  the  specific  developmental  issues  being  addressed  on  their  program  at  the  time  of  its 
participation  in  the  AWE,  the  decisions  to  be  made  from  gathered  data,  and  were  able  to 
specifically  tailor  their  program’s  activities  and  measures  in  the  AWE  to  support  these 
issues  and  decisions,  generally  reported  a  much  higher  level  of  perceived  benefit  from  the 
experiment. 

The  extent  to  which  acquisition  managers  implemented  the  DEL  in  the  TF  XXI 
AWE  can  be  measured  by  analyzing  the  extent  a  manager  was  able  to  tailor  or  influence  a 
program’s  specific  activities  in  the  AWE  to  reflect  the  program’s  current  acquisition  issues 
and  decisions.  Additionally,  one  can  analyze  the  level  at  which  AWE  data  was  derived 
and  used  to  support  future  acquisition  decisions. 

1.  The  Ability  to  Tailor 

Chapter  IV  illustrated  that  the  extent  a  manager  was  able  to  tailor  or  influence  a 
program’s  specific  activities  in  the  AWE  to  reflect  the  program’s  current  acquisition  issues 
and  decisions  had  a  direct  impact  on  the  extent  to  which  the  program  benefited  from 
participating.  Those  initiatives  reporting  a  high  level  of  ability  to  tailor  or  influence 
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generally  reported  much  higher  levels  of  perceived  benefit  from  participation  in  the 
AWE. 

The  level  that  acquisition  managers  were  able  to  participate  in  the  planning 
process  for  the  TF  XXI  AWE,  as  run  by  the  TRADOC  Joint  Venture  office,  varied  among 
programs.  It  is  unclear  if  the  level  of  participation  in  the  planning  process  for  specific 
programs  was  dictated  by  the  various  empowered  organizations  including  TRADOC, 
TRADOC  Analysis  Center  White  Sands  (TRAC-WSMR),  the  Operational  Evaluation 
Command  (OEC),  and  the  Test  and  Experimentation  Command  (TEXCOM)  or  if  the 
level  of  participation  was  a  conscious  decision  made  by  the  program  managers  and  user 
representatives  for  the  participating  initiatives. 

Many  of  the  surveyed  initiatives  reported  that  they  had  no  influence  on  then- 
system’s  role  in  the  experiment  or  on  the  analysis  plan  for  their  system.  One  program 
manager  stated  that  "there  was  a  hands  off  policy  instituted"  and  that  "there  was  no 
avenue  at  all  for  any  program  influence  [on  the  AWE  plan]."  Other  managers  reported 
that  they  had  no  real  influence  on  the  conduct  of  the  AWE  because  they  "had  no  issues  to 
be  addressed."  Another  manager  reported  that  "the  requirements  from  the  [user 
representative]  did  not  line  up  with  the  main  thrust  of  the  AWE,"  therefore  no  tailoring  of 
the  AWE  was  attempted.  Funding  shortages  also  contributed  to  the  limited  tailoring 
reported  by  mangers.  No  additional  funds  were  provided  to  any  program  specifically  to 
support  participation  in  the  AWE.  This  required  managers  to  "minimize  the  AWE’s 
impact  on  current  efforts  that  already  had  funds  obligated." 

The  inability  of  one  program  manager  to  influence  his  program’s  activities  in  the 
AWE  resulted  in  the  initiative  not  using  "real  communication  systems."  The  initiative 
was  not  "employed  over  realistic  distances."  As  a  result,  "times  and  completion  rates 
were  unrealistically  skewed."  The  initiative’s  level  of  benefit  from  the  AWE  was 
subsequently  reported  as  low.  A  user  representative  for  a  different  program  stated  that  he 
had  "little  influence  on  how  his  system  was  used  in  the  AWE"  and  this  "hampered  the 
[initiative’s]  ability  to  fully  participate."  Another  user  explained  that  he  was  unable  to 
influence  his  program’s  role  in  the  AWE  and  that  his  initiative  was  "artificially  restricted." 
The  result  was  a  failure  to  follow  establish  tactics,  techniques,  and  procedures  leading  to 
a  low  reported  value  for  the  gathered  data. 

Some  programs,  however,  reported  a  high  ability  to  tailor  or  influence  their 
initiative’s  participation  in  the  AWE.  One  user  representative  stated  that  they  were  "able 
to  tailor  all  of  the  required  activities"  because  they  were  "part  of  the  AWE  IPTs 
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[Integrated  Product  Teams]."  They  reported  that  they  played  an  active  role  in  developing 
the  analysis  plan  for  OPTEC  to  conduct  their  assessment.  They  also  assisted  in  the  data 
collection  and  assessment  process.  Another  user  representative  stated  that  his  office  was 
an  active  participant  "in  the  various  working  groups  by  the  primary  contractor,  PM,  and 
the  TSM  [TRADOC  System  Manager]  to  ensure  that  all  exercise  objectives  were  met  or 
exceeded." 

Generally,  those  initiatives  reporting  a  high  level  of  ability  to  tailor  or  influence 
reported  much  higher  levels  of  perceived  benefit  from  participation  in  the  AWE.  One 
program  manager  summarized  that  "we  pretty  much  had  our  way."  They  reported  their 
perceived  benefit  from  the  AWE  as  very  high.  The  initiative  that  participated  in  the 
AWE  IPTs  and  contributed  to  the  analysis  plan  discussed  earlier,  was  reportedly  praised 
by  the  then  TRADOC  commander  when  he  commented  that  the  system  was  a  "clear 
winner"  in  the  AWE. 

Both  program  managers  and  user  representatives  agree  that  to  achieve  maximuih 
benefit  from  AWEs,  acquisition  managers  must  be  able  to  participate  in  the  planning 
process.  One  manager  stated  that  acquisition  officials  should  participate  in  the  planning 
process  to  "define  clear  goals  and  objectives,  as  well  as  the  process  to  be  used  to  achieve 
the  goals  and  objectives."  Another  PM  stated  that  "a  study  should  be  conducted  to  define 
the  problems  to  be  resolved"  and  that  it  should  "include  a  section  on  the  perceived 
benefits  to  each  participant." 

The  TRADOC  Joint  Venture  office  has  recognized  the  value  of  having  program 
managers  participate  by  extending  their  role  in  the  planning  process  for  the  Joint 
Contingency  Force  (JCF)  AWE  to  be  held  in  late  1999.  A  program  manager  participating 
in  this  upcoming  AWE  stated  that  he  developed  Measures  of  Effectiveness  (MOEs)  for 
the  AWE  in  concert  with  independent  analysis  agencies.  For  the  JCF  AWE,  OPTEC 
representatives  will  meet  with  program  managers  to  determine  issues  that  are  linked  to 
AWE  objectives  [TRADOC  JCF- AWE,  1998].  Still,  no  additional  funding  is  provided  to 
the  program  manager  for  AWE  participation  or  for  implementation  of  any  derived 
recommendations.  Participation  in  AWEs  requires  expensive  prototyping,  manning, 
fielding,  training,  and  transportation  costs  that  must  be  drawn  from  existing  research  and 
development  accounts.  Program  managers  cannot  increase  their  roles  in  the  AWE 
process  and  ability  to  tailor  activities  without  dedicated  support  from  the  Army’s 
budgetary  process.  Therefore,  they  may  likely  continue  to  report  low  levels  of  benefit 
from  live  experiments. 
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2. 


Documentation  And  Use  Of  AWE  Data 


The  level  at  which  AWE  results  were  documented  within  the  acquisition 
management  office  had  a  direct  effect  on  the  ability  of  a  program  to  benefit  from 
participation.  As  discussed  in  Chapter  II  of  this  thesis,  the  value  of  any  evaluation  must 
be  judged  by  its  eventual  utility  in  implementing  change. 

A  significant  number  of  the  TF  XXI  AWE  initiatives  had  no  historic  data  or 
documentation  of  their  AWE  experience.  Several  programs  were  not  able  to  confirm  that 
their  system  participated  in  the  TF  XXI  AWE.  Six  of  the  original  seventy-two  offices 
initially  identified  to  participate  in  the  AWE  admitted  that  they  had  no  records  or 
knowledgeable  managers  that  could  comment  on  the  experiment.  Personnel  turnover  and 
attrition  can  account  for  some  of  this  absence  of  knowledge.  When  asked  to  list  specific 
issues  that  needed  to  be  investigated  in  the  AWE,  a  user  representative  stated 
"unfortunately,  we  are  not  aware  of  what  specific  developmental  issues  were  being 
addressed  on  the  program.  The  personnel  involved  in  them  at  the  time  have  PCS’d 
[moved  due  to  a  permanent  change  of  station]  and  that  data  is  not  available."  A 
substantial  majority  of  the  programs,  however,  had  significant  records  and  detailed  after¬ 
action  reports  from  the  AWE. 

A  respondent  discussed  the  AWE  documentation  process  and  the  eventual  utility 
of  the  data.  He  reported  that  "too  often  a  unit  pushes  to  get  a  system  in  an  AWE  and 
never  comes  into  the  decision  making  process  nor  does  guidance  get  passed  down  to  the 
people  who  have  to  execute  the  actions  to  make  it  happen."  Another  program  office 
admitted  their  shortcomings  by  stating  "we  needed  to  figure  out  a  better  way  to  document 
[lessons  learned  from  the  AWE]."  Another  manager  recommended  that  participants 
"document  as  well  as  possible  the  lessons  learned  so  that  a  third  party  can  understand." 

In  some  cases,  implementation  of  the  DEL  in  the  TF  XXI  AWE  was  limited  due 
to  poor  documentation  before  and  after  the  experiment.  In  many  cases,  issues  and 
decisions  to  be  made  were  not  identified  prior  to  the  AWE.  One  program  manager  stated 
that  his  system  "did  not  have  any  specific  developmental  issues  that  needed  to  be 
addressed  in  the  AWE."  Another  program  manager  stated  that  "no  specific  data  was 
proposed  to  be  gathered  or  evaluated,  nor  were  any  specific  issues  identified  for  analysis." 
This  lack  of  issue  identification  prevented  acquisition  managers  from  developing 
arguments  and  hypotheses  to  be  adjudicated.  Many  programs  did  not  perform  analysis  or 
develop  measures  to  implement  decisions.  One  program  manager  stated  repeatedly  that 
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"the  recommendations  fell  outside  the  responsibility  of  the  PMO  [Program  Management 
Office]."  Another  program  manager  stated  "to  date,  we  have  not  seen  the  [AWE]  report, 
and  have  only  heard  rumors  as  to  the  outcome."  The  manager  continued  that  "there  was 
no  benefit  to  the  program  from  the  AWE." 

A  manager  from  an  initiative  reporting  a  high  level  of  perceived  benefit 
recommended  that  AWE  participants  "define  clear  goals  and  objectives,  as  well  as  the 
process  to  be  used  to  achieve  the  goals  and  objectives."  Another  program  reporting  very 
high  benefit,  contributed  the  benefit  to  "thorough  planning  and  understanding  of  what  the 
expected  outcomes  were  supposed  to  be."  A  program  with  a  low  level  of  reported  benefit 
explained  that  their  problems  related  to  a  failure  to  identify  how  collected  data  would  "be 
used  to  focus  future  activity." 

One  program  manager  effectively  summarized  the  requirement  for  documentation 
and  planning  by  stating  "it  is  important  for  a  program  manager  to  develop  test  objectives 
and  measurement  processes  that  identify  product  improvements,  areas  for  future  research, 
and  follow-on  programming,  while  allowing  AWE  results  to  drive  go/no-go  decisions  for 
the  Army." 


3.  Conclusions  on  the  Decision  to  Experiment  Ladder 

The  data  clearly  indicates  that  the  benefit  derived  from  participation  in  AWEs  can 
be  increased  by  managers  developing  detailed  experiment  objectives  and  expected 
outcomes,  systems  for  data  documentation  and  analysis,  and  strategies  for  implementation 
of  AWE  data  and  recommendations.  Additionally,  a  program’s  specific  AWE  activities 
should  be  tailored  to  reflect  these  objectives,  systems,  and  strategies. 

E.  PROGRAM  MATURITY 

As  illustrated  in  Chapter  IV,  a  relationship  exists  between  an  initiative’s 
programmatic  maturity  and  the  benefit  gained  from  participating  in  the  AWE.  More 
maturity  does  not  directly  transcribe  into  a  higher  ability  to  benefit,  but  rather  a  point 
exists  where  a  program  is  best  positioned  to  benefit  from  experimentation.  All  of  the 
collected  objective  data  supports  the  finding  that  an  initiative  in  the  mid-developmental 
phases  of  acquisition  will  be  best  positioned  to  benefit  from  AWE  participation. 

Acquisition  managers  agree  that  timing  is  critical  to  success  in  an  AWE. 
Programs  in  Concept  Exploration  (CE)  are  too  immature  to  effectively  participate. 
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Programs  in  production  caimot  take  advantage  most  findings  or  recommendations  fi'om 
the  AWE.  Even  OPTEC  agrees  that  "AWE  planners  should  carefully  consider  whether  it 
is  worthwhile  to  devote  scarce  data  collection  and  assessment  resources  to  assess  post- 
Milestone  III  systems"  [ECC,  1997]. 

A  key  factor  of  maturity,  as  defined  by  survey  participants,  was  system  reliability. 
Immature  initiatives  were  generally  not  environmentally  hardened  or  ruggedized  for  field 
trials.  Poor  operator  handling  of  these  non-ruggedized  systems  resulted  in  excessive 
maintenance  problems  that  were  reported  as  system  failures  and  contributed  to  a  negative 
post-AWE  analysis.  A  program  manager  commented  that  the  AWE  was  "not  a  good 
environment  for  marginal  systems."  According  to  this  manager,  if  the  system  is  not 
sufficiently  mature,  "the  opportunity  for  failure  is  much  greater  than  the  opportunity  for 
success." 

Participating  initiatives  must  also  be  sufficiently  mature,  programmatically,  so 
that  prototypes  and  demonstration  versions  are  readily  available  for  participation.  In 
several  cases,  surrogate  systems  were  included  in  the  AWE  to  simulate  proposed 
capabilities.  Often  these  surrogates  did  not  effectively  represent  proposed  production 
systems.  A  program  manager  stated  that  immature  systems  were  responsible  for 
"reducing  the  effectiveness  or  otherwise  limiting  the  lessons  learned  by  other  platforms." 
Respondent  comments  show  that  the  varying  levels  of  maturity  and  functionality  of 
initiatives  affected  the  overall  performance  of  the  experimental  force."  It  was  stated  that 
"soldiers  are  practical  and  will  not  waste  time  using  technology  that  is  cumbersome  or 
does  not  work  As  a  result,  substandard  systems  were  ignored." 

Some  mature  programs  that  had  moved  into  production  prior  to  the  AWE  were 
able  to  benefit  from  refinement  of  Tactics,  Techniques,  and  Procedures  (TTP)  at  a  higher 
level  then  immature  programs.  Production  items  generally  provided  the  reliability 
required  to  adequately  stress  the  system  using  realistic  methods.  Program  managers 
agreed  that  systems  providing  "new  capabilities"  or  "leap-ahead  technologies"  were  best 
situated  to  benefit  from  the  development  of  TTPs. 

Program  managers  often  lacked  the  funding  required  to  implement  changes 
recommended  by  the  AWE  analysis.  One  program  manager  stated  that  "programs  must 
be  sufficiently  mature  to  preclude  the  need  for  drastic  changes  as  a  result  of  the  AWE." 
As  stated  earlier,  one  initiative  was  forced  to  make  "major  changes  in  system 
architecture"  as  a  result  of  the  AWE.  Funding  for  this  restructuring  was  made  available  at 
the  expense  of  other  programs  within  the  program  management  office. 
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Survey  comments  show  that  programs  at  any  level  of  maturity  can  benefit  from 
AWE  participation.  Programs  that  are  early  in  development,  such  as  CE  and  PDRR,  can 
demonstrate  the  benefits  of  new  concept  technologies  to  the  force,  and  gain  subsequent 
program  support.  Mature  programs  that  are  in  production,  can  gain  visibility  and  positive 
marketing  aspects  required  for  continued  funding.  However,  initiatives  in  mid- 
developmental  phases  of  acquisition,  that  are  sufficiently  mature  to  tolerate  the  harsh 
environment  of  a  live  experiment  but  do  not  have  a  finalized  architecture,  seem  to  benefit 
most  from  an  AWE. 

F.  RISK  AND  BENEFIT  ANALYSIS 

Survey  comments  indicate  a  substantial  risk  to  programs  participating  in  an  AWE. 
Program  risks  include  factors  beyond  the  costs  of  participation,  to  include  a  poor  return 
on  investment,  potential  negative  exposure,  and  extensive  changes  in  requirements. 
Acquisition  managers  contemplating  participation  in  future  AWEs  could  benefit  from 
conducting  a  detailed  analysis  of  the  risks  associated  with  AWEs,  comparing  these  risks 
to  potential  benefits,  and  considering  their  tolerance  for  risk  as  a  factor  in  making  a 
decision  to  pa^cipate. 

Program  managers  reported  a  wide  range  of  tolerance  for  risk.  One  program 
manager  stated  "we  had  nothing  to  lose."  The  program  had  no  funding  for  future 
production  and  the  program  manager  believed  that  the  AWE  served  as  a  "last  chance" 
effort.  Reportedly,  if  the  initiative  could  have  shown  merit  at  the  AWE  then  participation 
in  the  experiment  would  have  been  a  good  investment.  Another  manager  stated  "we  had 
nothing  to  gain."  The  design  of  the  system  was  finalized  and  a  production  decision  had 
been  made  and  funded. 

Data  collected  from  TF  XXI  AWE  participants  can  be  used  to  measure  the  risks 
and  benefits  involved  with  AWE  participation.  This  thesis  presents  a  model  to  measure 
risk  as  compared  to  potential  benefit.  Although  this  model  has  not  been  validated 
through  actual  implementation,  it  is  supported  by  the  findings  of  this  thesis.  Program 
managers  should  complete  this  risk  and  benefit  analysis,  and  if  necessary,  modify  the  risk 
analysis  plan  based  on  the  unique  situation  of  their  program  and  their  personal  tolerance 
for  risk.  Potential  risks  involved  with  AWE  participation  are  presented,  followed  by  a 
description  of  risk  and  benefit  factors.  A  method  to  compare  risk  factors  and  potential 
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benefits  and  analyze  the  risk  situation  for  AWE  initiatives  is  illustrated.  Finally,  methods 
to  mitigate  some  of  this  risk  are  presented. 

1.  Potential  Risks 

Potential  risks  are  areas  where  acquisition  programs  may  be  negatively  effected. 
This  thesis  has  reveled  three  significant  risks  involved  with  participation:  poor  return  on 
investment,  potential  negative  exposure,  and  extensive  change  in  requirements. 

a.  Poor  Return  on  Investment 

TF  XXI  AWE  acquisition  managers  stressed  that  AWE  participation  is 
resource  intensive.  One  manager  stated  that  "PMs  must  determine  if  potential  return 
outweighs  the  cost  in  prototyping,  manning,  fielding,  transportation,  etc."  He  stated  that 
the  Army  should  provide  funding  to  program  offices  for  participation.  However,  no 
funding  has  been  provided  for  past  AWEs  and  as  stated  in  the  plan  for  the  upcoming  Joint 
Contingency  Force  (JCF)  AWE,  submitting  agencies  must  "bear  the  complete  costs 
associated  with  the  initiative"  [TRADOC  JCF-AWE,  1998,  p3].  Given  the  low  levels  of 
benefit  reported  by  some  programs,  there  is  an  inherent  risk  that  the  benefits  received  by 
any  one  initiative  will  not  justify  the  cost. 

b.  Potential  Negative  Exposure 

Program  exposure  to  both  user  and  senior  Army  leadership  and  the 
associated  marketing  effects  of  AWE  participation  must  be  considered  with  both  then- 
positive  and  negative  potential  consequences.  Some  initiatives  with  outstanding  AWE 
performance  received  high  acclaim  and  significant  support  for  future  funding  decisions. 
However,  this  thesis  has  many  examples  of  programs  that  received  negative  feedback 
from  the  AWE.  The  results  of  the  AWE  provided  negative  marketing  for  some  programs 
that  hampered  their  ability  to  move  into  production.  One  initiative  had  two  of  four 
systems  fail  at  the  AWE.  This  was  reported  as  a  reliability  problem  when  damage  was 
likely  due  to  poor  operator  handling.  Another  system  received  a  poor  evaluation 
primarily  due  to  host  vehicle  failures.  The  risk  of  program  exposure,  positive  or  negative, 
must  be  considered  prior  to  AWE  participation. 
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c.  Excessive  Changes  in  Requirements 

One  of  the  stated  goals  of  the  TF  XXI  AWE  was  to  provided  information 
to  support  investment  decisions  and  refinement  of  requirements  [ECC,  1996].  Some  TF 
XXI  initiatives  received  so  much  information  that  the  refined  requirements  negated  the 
current  system  design.  These  programs  were  then  either  terminated,  or  underwent 
complete  restructuring.  One  such  program  was  in  production  at  the  time  of  the  AWE  and 
subsequently  was  forced  to  design  a  complete  new  architecture  and  develop  new  metrics 
to  measure  performance.  Although  these  changes  may  represent  an  ultimate  benefit  from 
AWE  participation,  some  acquisition  programs  may  not  be  positioned  to  adequately  adapt 
to  such  drastic  changes  in  requirements. 

2.  Risk  Factor  Ratings 

Risk  factors  are  those  issues  that  can  be  measured  by  an  acquisition  manager  in  an 
attempt  to  determine  overall  risk  from  participation.  As  shown  in  Table  2,  these  risk 
factors  directly  contribute  to  the  three  identified  risks  of  poor  return  on  investment, 
negative  exposure,  and  excessive  changes  in  requirements.  This  thesis  has  revealed  seven 
contributing  risk  factors  to  AWE  participation  including:  (1)  system  maturity,  (2) 
ruggedization  and  maintainability,  (3)  how  the  system  fits  within  the  focus  of  AWE,  (4) 
funding  availability,  (5)  equipment  availability,  (6)  incoming  biases,  and  (7)  the  status  of 
production  decisions. 


Table  2.  Factors  Contributing  to  Potential  Risks 


Poor  Return  on 
Investment 

Maturity,  Focus  of  the  AWE,  Funding  Availability, 
Equipment  Availability, 

Potential  Negative 
Exposure 

Maturity,  Ruggedization,  Funding  Availability, 
Equipment  Availability,  Incoming  Biases 

Excessive  Changes 
in  Requirements 

Maturity,  Funding  Availability,  Status  of  Production 
Decisions 

a.  Maturity 

Maturity  is  measured  in  terms  of  the  system’s  programmatic  stage  of 
development,  from  Concept  Exploration  to  Production.  Maturity  effects  the  ability  of  a 
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program  to  perform  adequately  as  well  as  the  ability  of  the  initiative  to  implement 
recommendations  derived  from  the  AWE.  Programs  in  Concept  Exploration  are  high  risk 
because  of  their  unpredictability  and  vaguely  defined  roles  and  requirements.  Programs 
in  Production  or  Limited  Production  are  considered  medium  risk  because  they  may  be  too 
far  in  development  to  capitalize  on  recommendations.  Programs  in  PDRR  or  EMD  are 
considered  low  risk. 

b.  Ruggedization  and  Maintainability 

Ruggedization  and  maintainability  are  measured  as  high  or  low.  A  system 
that  is  highly  mgged  is  able  to  withstand  the  stresses  associated  with  operational  use  in 
harsh  environments.  A  system  that  scores  low  in  this  factor  has  sensitive  components 
with  maintenance  procedures  that  are  difficult  to  conduct  in  a  field  environment.  Low 
mggedization  may  be  associated  with  early  prototype  systems. 

c.  Focus  of  the  AWE 

A  manager  must  analyze  the  published  goals  and  objectives  of  the  AWE  as 
stipulated  by  the  planning  officials  and  characterize  how  his  program  fits  within  the 
experiment  focus.  Also,  a  manager  must  measure  how  the  AWE  will  address  the  issues 
facing  his  initiative.  How  an  initiative  fits  into  the  focus  of  the  AWE  is  measured  as 
complimentary  or  ancillary.  As  demonstrated  in  the  TF  XXI  AWE,  those  programs  that 
are  not  a  high  priority  of  the  analyzing  agency  may  not  receive  adequate  focus  and 
feedback.  This  inadequate  focus  and  feedback  can  lead  to  a  poor  return  on  investment. 

d.  Funding  Availability 

Participation  in  an  AWE  is  resource  intensive.  A  program  must  have 
sufficient  funds  for  prototypes,  manning,  training,  transportation,  and  maintenance. 
Required  funding  will  vary  based  on  the  complexity  of  the  system.  Additional  funds  will 
not  normally  be  available  specifically  for  AWE  participation.  Low  availability  of  funding 
can  contribute  to  an  inability  to  properly  prepare  for  the  AWE  and  an  inability  to 
implement  any  derived  findings.  This  can  result  in  negative  exposure  and  low  return  on 
investment.  Funding  is  measured  as  sufficient  or  insufficient.  Sufficiently  funded 
programs  will  have  budgeted  dollars  specifically  for  AWE  participation  and  are 


77 


considered  low  risk.  Insufficiently  funded  programs  will  have  tight  budgetary  constraints 
and  are  considered  high  risk. 

e.  Equipment  AvaUabilUy 

Development  of  prototypes  or  surrogates  for  AWE  participation  requires 
additional  costs,  time,  and  developmental  testing  beyond  that  normally  experienced  in 
system  development.  Surrogates  and  prototypes  may  also  be  less  robust  and  contribute  to 
poor  performance  at  AWEs,  potential  negative  exposure,  and  low  return  on  investment. 
Mature  systems  with  readily  available  production  items  are  best  positioned  for  AWE 
participation.  Equipment  availability  is  measured  on  a  continuum  from  production  items 
to  surrogate  representatives.  Initiatives  with  readily  available  production  models  are 
considered  low  risk.  Initiatives  that  require  the  development  of  prototypes  or  surrogates 
are  considered  high  risk. 

f.  Incoming  Biases 

Program  managers  from  the  TF  XXI  AWE  stated  that  incoming  biases  by 
users,  evaluators,  and  Army  leadership  can  prevent  any  real  analysis  of  a  system’s 
performance  at  the  AWE.  Biases  are  measured  as  either  favorable  or  unfavorable. 
Unfavorable  biases  can  contribute  to  reported  poor  performance  and  negative  exposure. 
Those  programs  with  unfavorable  biases  are  considered  high  risk.  Those  programs  with 
favorable  biases  are  considered  low  risk. 

g.  Status  of  Production  Decisions 

Production  decision  status  is  measured  as  either  pre-production  with 
Milestone  HI  decision,  inmiinent  production  decision,  or  post  production.  Programs  that 
have  secured  approval  and  funding  for  production  prior  to  participation  in  an  AWE  but 
have  not  yet  begun  production,  face  the  added  risk  of  poor  performance  and  loss  of 
support.  These  pre-production  decision  initiatives  are  categorized  as  high  risk.  Those 
initiatives  with  imminent  production  decisions  are  considered  medium  risk,  in  that  AWE 
performance  can  significantly  influence  the  survivability  of  the  system.  Programs  in  post¬ 
production  or  with  no  pending  production  decision  are  categorized  as  low  risk. 


3. 


Benefit  Factor  Ratings 


Benefit  factors  are  those  issues  that  can  be  measured  by  an  acquisition  manager  in 
an  attempt  to  determine  the  potential  to  benefit  from  AWE  participation.  These  factors 
can  be  measured  in  terms  of  potential  to  benefit  based  on  the  unique  needs  and  situation 
of  the  initiative  in  question.  This  thesis  has  revealed  seven  benefit  factors  to  AWE 
participation  including:  (1)  marketing  and  exposure  of  program,  (2)  early  user  feedback, 
(3)  refinement  of  user  requirements,  (4)  development  of  tactics,  techniques,  and 
procedures  (TTP),  (5)  follow-on  support  for  funding  and  production  decisions,  (6) 
information  on  integration,  interfaces,  and  interoperability,  (7)  exposure  of  developers  to 
the  user’s  environment. 

a.  Marketing  and  Exposure  of  Program 

The  effects  of  marketing  and  exposure  of  a  program  are  a  result  of  system 
performance  at  the  AWE  and  cannot  be  accurately  predicted.  The  potential  for  benefit 
from  advantageous  exposure  is  measure  based  on  needs  of  the  program.  A  program  that 
requires  positive  exposure  for  survival,  would  rate  this  area  as  high  potential  benefit. 
Those  programs  that  do  not  require  this  marketing  effect,  or  already  have  strong  support 
among  senior  leadership,  would  rate  this  area  as  low  potential. 

b.  Early  User  Feedback 

The  potential  to  benefit  from  early  user  feedback  is  measured  based  on 
needs  of  the  program.  Those  programs  that  have  had  little  to  no  user  feedback  would  rate 
this  area  as  high.  Those  programs  that  already  possess  sufficient  feedback,  or  those  that 
are  not  in  a  position  to  implement  any  recommendations  from  the  feedback,  would  rate 
this  area  as  low  potential. 

c.  Refinement  of  User  Requirements 

The  potential  to  benefit  from  refinement  of  requirements  is  measured 
based  on  the  initiative's  stage  of  development.  Those  programs  early  in  Concept 
Exploration  would  score  this  area  as  high,  because  they  are  best  positioned  to  implement 
the  refinements  without  a  significant  investment.  EMD  programs  would  score  medium. 
Programs  late  in  development  or  into  production  would  score  this  area  as  having  low 
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potential  benefit  because  the  refinements  caimot  be  adequately  implemented  in 
production  models. 

d.  Development  of  Tactics,  Techniques,  and  Procedures  (TTP) 

The  potential  to  benefit  from  TTP  development  is  measured  based  on 
needs  of  the  program.  Initiatives  that  are  new  concept  technologies  with  no  current  TTP 
would  rate  this  area  as  high.  While  existing  TTPs  can  be  modified  and  refined  based  on 
AWE  data,  programs  that  are  not  new  concepts  to  the  Army  and  have  established  TTPs 
do  not  require  this  data  to  move  into  production.  These  initiatives  would  rate  this  area  as 
low  potential  benefit. 

e.  FolloW‘On  Support  for  Funding  and  Production  Decisions 

The  need  for  funding  and  support  for  production  decisions  determines  the 
potential  benefit  in  this  area.  Those  programs  with  production  decisions  pending  would 
rate  this  potential  benefit  as  high,  as  favorable  AWE  performance  can  result  in  favorable 
decisions  by  Army  leadership.  Those  programs  with  future  production  decisions  would 
rate  this  area  as  medium,  as  the  derived  AWE  data  can  be  used  to  support  these  decisions. 
Programs  that  are  currently  in  production  have  no  extensive  need  for  funding  support  and 
would  rate  this  as  a  low  potential  benefit. 

/.  Information  on  Integration,  Interfaces,  and  Interoperability 

Potential  benefit  in  this  area  is  measured  based  on  need.  Those  systems 
with  a  high  requirement  to  integrate  or  interface  with  other  programs  would  rate  this  area 
as  high.  An  initiative  that  does  not  interface  or  interoperate  with  other  systems  would 
score  this  as  a  low  potential  benefit. 

g.  Exposure  of  Developers  to  the  User’s  Environment 

The  exposure  of  developers  to  the  user’s  environment  is  rated  as  a  high 
potential  benefit  for  those  programs  where  managers  and  engineers  have  little  field 
experience.  Those  programs  with  managers  and  engineers  who  have  sufficient  exposure 
to  the  user’s  environment  would  rate  this  potential  benefit  as  low. 
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4. 


Measurement  and  Analysis  of  Risk  and  Benefit 


a.  Risk  Measurement 

The  presented  risk  factors  can  be  subjectively  measured  and  totaled  using 
the  risk  analysis  matrix  shown  as  Table  3.  In  Table  3,  an  acquisition  manager  should 
subjectively  measure  each  of  the  seven  risk  factors  as  either  high,  medium,  or  low,  and 
circle  the  appropriate  rating.  The  total  number  of  ratings  for  each  of  the  three 
measurements  should  be  totaled  and  entered  as  the  factor  total.  Each  of  the  three  factor 
totals  should  then  be  multiplied  by  a  risk  multiplier  with  the  high  risk  factor  total 
multiplied  by  three,  the  medium  risk  factor  total  multiplied  by  two,  and  the  low  risk  factor 
total  multiplied  by  one.  The  results  of  this  multiplication  should  then  be  entered  as  the 
risk  total  for  each  of  the  three  measurement  levels.  All  three  risk  totals  should  be  added 
together  to  compute  the  total  risk  score. 

The  results  of  a  hypothetical  AWE  initiative  is  presented  in  Table  4  as  a 
risk  factor  example.  This  hypothetical  initiative  is  a  new  digital  communications  system. 
This  initiative  is  currently  in  the  PDRR  phase  of  development,  so  maturity  is  assessed  as 
low  risk.  The  system  has  not  been  extensively  ruggedized  but  has  low  maintenance 
requirements,  therefore,  ruggedization  /  maintainability  is  assessed  as  moderate  risk.  The 
focus  of  the  AWE  is  digital  communication  systems  so  the  initiative  is  considered 
complementary  and  risk  is  assessed  as  low.  Funding  is  available  for  research  and 
development  on  this  item,  though  specific  funds  have  not  been  programmed  for  the 
AWE.  Therefore,  the  funding  availability  risk  factor  is  assessed  as  moderate.  A 
prototype  of  the  system  is  being  developed  specifically  for  the  AWE,  so  the  equipment 
availability  risk  factor  is  rated  as  high.  The  Army  leadership  believes  that  this 
communication  system  is  of  paramount  importance  to  the  future  of  army  digitization  and 
has  a  high  level  of  favorable  biases.  Therefore,  the  bias  risk  factor  is  assessed  as  low. 
The  program  believes  it  will  move  into  EMD  shortly  and  be  facing  a  production  decision 
within  the  next  two  years,  and  thus  has  assessed  the  production  decision  risk  factor  as 
medium.  The  risk  factor  totals  equate  to  one  high  risk  factor,  three  medium  risk  factors, 
and  three  low  risk  factors.  The  risk  multipliers  are  applied  for  a  total  risk  score  of  12. 
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Table  3.  Risk  Factor  Computation 
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Table  4.  Risk  Factor  Example 


Factor 
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b.  Benefit  Measurement 

Potential  to  derive  benefit  from  the  AWE  can  be  measured  based  on  the 
factors  presented  earlier  in  this  chapter  and  totaled  using  Table  5.  In  Table  5,  an 
acquisition  manager  should  subjectively  measure  each  of  the  seven  potential  benefit 
factors  as  either  high,  medium,  or  low,  and  circle  the  appropriate  rating.  The  total 
number  of  ratings  for  each  of  the  three  measurements  should  be  totaled  and  entered  as  the 
factor  total.  Each  of  the  three  factor  totals  should  then  be  multiplied  by  a  benefit 
multiplier  with  the  high  benefit  factor  total  multiplied  by  three,  the  medium  benefit  factor 
total  multiplied  by  two,  and  the  low  benefit  factor  total  multiplied  by  one.  The  results  of 
this  multiplication  should  then  be  entered  as  the  benefit  total  for  each  of  the  three 
measurement  levels.  All  three  benefit  totals  should  be  added  together  to  compute  the 
total  benefit  score. 
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Table  5.  Benefit  Factor  Computation 
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The  results  of  a  benefit  factor  analysis  of  a  hypothetical  AWE  initiative  is 
presented  as  Table  6.  This  example  uses  the  same  hypothetical  initiative  presented  above, 
in  risk  factor  computation.  The  program  manager  has  assessed  his  potential  to  benefit  in 
each  of  the  seven  benefit  factors.  The  program  manager  has  determined  that  his  program 
has  a  moderate  need  for  marketing  and  exposure.  This  program  does  not  have  a  wide 
familiarity  among  Army  leadership,  yet  this  exposure  is  not  required  for  program  survival 
in  its  current  phase.  Therefore,  he  has  assessed  the  potential  for  a  marketing  and 
exposure  benefit  as  medium.  The  program  manager  has  determined  that  his  initiative 
requires  early  user  feedback  from  its  user  community  and  has  assessed  the  potential  to 
benefit  from  in  this  area  as  high.  The  initiative  is  currently  in  PDRR  and  has  a  medium 
potential  to  benefit  from  requirement  refinement  while  at  the  AWE.  While  some  TTPs 
exist  for  digital  communications  system,  the  program  manager  has  determined  that  they 
have  a  moderate  need  to  refine  these  TTPs  and  has  assessed  the  potential  to  benefit  in  this 
area  as  medium.  The  program  manager  believes  the  initiative  will  move  into  EMD 
shortly  and  be  facing  a  production  decision  within  the  next  two  years,  and  thus  has 
assessed  the  potential  to  benefit  from  funding  support  and  future  production  decisions  as 
medium.  Being  a  digital  communications  system,  the  program  requires  detailed 
information  on  integration,  interfaces,  and  interoperability,  and  thus  has  assessed  this 
potential  benefit  as  high.  The  program  managers  and  engineers  currently  have  little 
exposure  to  the  user’s  field  environment,  so  this  factor  has  also  been  assessed  as  having  a 
high  potential  benefit.  The  results  of  the  benefit  assessment  included  three  areas  of 
potentially  high  benefit,  four  areas  of  medium  benefit,  and  one  area  with  low  potential 
benefit  to  the  program.  With  the  benefit  multipliers  applied,  the  total  benefit  score 
computed  to  seventeen. 

c.  Comparison  and  Analysis 

Resulting  risk  and  potential  benefit  measurements  can  be  compared  using 
the  risk  analysis  chart  illustrated  below  as  Figure  22.  When  computed  risk  and  benefit 
scores  are  compared,  four  risk  analysis  quadrants  are  developed.  Quadrant  I  describes  an 
initiative  with  a  high  potential  to  benefit  from  AWE  participation  with  very  low  risk  to 
the  program.  Program  managers  assessing  their  system  in  Quadrant  I  are  in  the  best 
position  to  benefit  from  AWEs.  Quadrant  HI  describes  a  program  with  very  low  potential 
benefit  and  high  associated  risk.  From  an  acquisition  manager’s  perspective,  these 
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programs  would  be  best  served  by  not  participating  in  AWEs.  Quadrant  n  and  Quadrant 
IV  describe  programs  with  situations  that  are  not  as  clear  as  the  previous  two  cases.  A 
program  in  Quadrant  n  has  a  high  potential  for  benefit  but  not  without  significant  risk.  In 
this  quadrant,  an  acquisition  manager  must  determine  his  tolerance  for  risk  in  making  an 
AWE  participation  decision.  A  program  in  Quadrant  IV  has  low  risk  from  participation 
but  also  does  not  stand  to  receive  significant  benefit.  Program  managers  in  this  situation 
should  base  their  participation  decision  on  a  monetary  basis  or  consider  other  potential 
benefits  not  discussed  in  this  thesis. 


Table  6.  Benefit  Factor  Example 
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Feedback 

Early  in  Concept  ^ 
Development 

Engineering  Phase  of 

^  Late  in  Development  / 
Production 

TIP  Development 

No  Current  TTPs  /  New^ 
Concent  Technolosv 
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^  TTPs  Established  /  No 

New  Concents 

j^ppoit  for  Fnndins  / 
Ifrodncfibn  Dedsions 

^Currently  in  Production 

Info  on  Integration/  { 
Ihter&ces/ 

InferoneraliUitv  _ 

Moderate  Need 

System  Does  Not 

Interface  /  Interoperate 

Eimmmte  ^  User^s 
Envfroihnait 

^Managers  /  Engineers^ 
Little  Exposui^^ 

^  Moderate  Need 

Field  Experienced 

Managers  and 

Engineers 

FactorTotaJs _ 

3 

4 

0 

3 

2 

1 

9 

.  8 

0 

Ben^t  (Total  Of 

17 
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The  results  of  a  risk  and  benefit  comparison  and  analysis  for  a  hypothetical 
AWE  initiative  are  shown  in  Figure  23.  The  hypothetical  initiative  is  the  same 
communication  system  used  above  for  the  risk  and  benefit  factor  measurement.  The 
system’s  total  risk  score  was  twelve  and  the  total  benefit  score  was  seventeen.  As  shown 
in  Figure  23,  the  resulting  comparison  lies  in  Quadrant  II  of  the  risk  /  benefit  comparison 
chart.  This  signifies  that  the  initiative  has  high  potential  benefit  from  AWE,  but  with 
moderately  high  risk.  Based  on  this  analysis,  the  initiative’s  acquisition  manager  must 
determine  his  tolerance  for  risk  in  making  an  AWE  participation  decision.  Through  the 
application  of  risk  mitigation,  as  discussed  below,  the  risk  score  can  be  reduced  so  that 
the  initiative  lies  within  Quadrant  I,  where  potential  benefits  are  high,  but  participation 
risk  is  low. 


^1.  - 

Potential  in  <  - 

Quad  I 

High  Potential 

Benefit  /  Low 

Risk 

Quad  n 

High  Potential 
Benefit  /  High 

Risk 

Benefit  Score 

Quad  rv 

Quad  in 

Low  Potential 

Low  Potential 

Benefit  /  Low 

Benefit  /  High 

0  - 

Risk 

Risk 

0  10.5  21 

Risk  Score 

Figure  22.  Risk  /  Benefit  Comparison  and  Analysis 
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21 

(2)- 


Potential  10  5 

Benefit  Score 


0  21 

Risk  Score 

Figure  23.  Risk  /  Benefit  Comparison  and  Analysis  Example 

5.  Risk  Mitigation 

The  risks  discussed  previously  in  this  chapter  are  based  on  reported  results  of  the 
TF  XXI  AWE.  Programs  participating  in  future  AWEs  can  mitigate  these  risks  based  on 
the  TF  XXI  experience.  Methods  to  mitigate  risk  include  program  budgeting,  marketing 
to  the  user,  user  training,  and  participation  in  the  planning  process. 


Quad  I 

High  Potential 

Benefit  /  Low 

Risk 

Quad  n 

High  Potential 

1  Benefit  /  High 

j  Risk 

1  „  . . . 

Quad  IV 

Low  Potential 

Benefit  /  Low 

Risk 

1 

1  Quad  m 

j  Low  Potential 

1  Benefit  /  High 
j  Risk 

_ 1 _ 

a.  Program  Budgeting 

The  experiment  plan  for  the  JCF-AWE  states  that  one  the  main  criteria 
used  for  consideration  for  inclusion  is  "the  ability  of  the  submitting  agency  to  bear  the 
complete  costs  associated  with  the  initiative,  to  include  its  integration  into  the  JCF-AWE 
architecture"  [TRADOC  JCF-AWE,  1998,  p3].  A  manager  participating  in  the  TF  XXI 
AWE  stated  that  he  needed  to  conduct  "monetary  trade-offs  between  supporting  his 
current  noission  and  supporting  the  AWE."  Certainly,  program  managers  do  not  need  to 
be  the  sole  source  for  AWE  funding.  Stakeholders  in  the  system,  to  include  program 
managers,  combat  developers,  TRADOC  system  managers,  and  battle  labs  can  all 
combine  resources  to  fund  AWE  participation.  AWE  participation  risks  centered  on 
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funding  issues  can  be  mitigated  by  proactively  budgeting  for  AWE  participation  early  in 
the  system’s  lifecycle  using  available  research  and  development  dollars.  This  proactive 
budgeting  will  help  ensure  that  sufficient  funds  are  available  for  AWE  participation 
without  the  need  to  conduct  schedule  or  performance  trade-offs.  Certainly,  as  the  likely 
benefit  of  AWE  participation  can  be  enhanced,  program  managers  will  be  more  motivated 
to  build  participation  in  their  program  planning. 

b.  Marketing  to  User 

In  several  cases,  initiative  managers  stated  that  their  system  received  a 
poor  assessment  at  the  AWE  because  actual  users  failed  to  appreciate  the  capabilities  of 
the  system.  One  such  initiative  reported  that  it  did  not  get  much  use  at  the  AWE  because 
the  soldiers  were  concerned  with  some  potential  negative  characteristics  of  its  design.  He 
stated  that  the  soldiers  using  the  equipment  did  not  fully  understand  these  characteristics 
and  their  effects.  Marketing  the  advantages  of  the  system  to  the  soldier  to  enhance  the 
perception  of  the  initiative  would  have  increased  opportunities  to  collect  data  during  the 
AW^.  Risks  associated  with  preconceived  biases  by  users,  evaluators,  and  Army 
leadership  can  be  mitigated  with  an  early  investment  in  system  marketing. 

c.  User  Training 

One  of  the  most  often  commented  factor  contributing  to  a  program’s 
success  at  the  AWE  was  the  quality  of  user  skills.  One  respondent  stated  that  there  were 
"too  many  new  systems  being  introduced  in  the  AWE  and  it  confused  the  soldiers." 
Reportedly,  soldiers  were  "overwhelmed  at  the  complexity  of  the  additional  systems 
installed  on  their  vehicles."  Another  acquisition  manager  stated  that  "in  many  cases  the 
systems  under  evaluation  were  simply  turned  off  or  never  turned  on."  All  new  systems 
preparing  for  fielding  are  required  to  develop  initial  fielding  training  plans.  Normally  this 
consists  of  civilian  contractors  training  senior  military  instructors,  who  in  turn  train  the 
actual  user.  Systems  that  are  early  in  their  development  lifecycle  may  not  have  training 
programs  designed  or  tested.  Risks  associated  with  maturity,  ruggedization, 
maintainability,  and  biases  can  be  mitigated  through  high  quality,  hands-on  training  of 
actual  users  in  field  conditions,  prior  to  AWE  participation. 
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d.  Participation  in  Planning  Process 

As  stated  earlier  in  this  thesis,  the  extent  that  a  manager  was  able  to  tailor 
or  influence  a  program’s  specific  activities  in  the  AWE  to  relate  to  the  program’s 
acquisition  issues  and  decisions  contributed  to  the  extent  that  the  program  was  able  to 
benefit.  The  only  way  to  tailor  or  influence  a  program’s  activities  in  the  AWE  is  by 
investing  heavily  in  the  AWE  planning  process.  Managers  should  attempt  to  influence 
their  system’s  role  in  the  AWE,  the  scenarios  that  it  is  placed  in,  the  analysis  and  data 
collection  plan,  and  the  tactics,  techniques,  and  procedures  to  be  implemented  by  the  user. 
AWE  officials  will  often  call  upon  user  representatives  to  provide  Subject  Matter  Experts 
(SMEs)  to  aid  in  the  data  collection  effort.  Risks  associated  with  the  focus  of  the  AWE, 
equipment  availability,  equipment  maintainability,  and  preconceived  biases  can  be 
mitigated  by  active  participation  in  the  AWE  planning  process. 

G.  CHAPTER  SUMMARY 

This  chapter  presented  analysis  and  findings  on  the  contributing  factors  to 
acquisition  managers  benefiting  from  Advanced  Warfighting  Experiments.  The  Task 
Force  XXI  AWE  was  limited  in  its  ability  to  provide  benefit  to  participating  acquisition 
managers  due  the  nature  of  the  experiment.  Limitations  in  repeatability,  data  collection, 
interactions,  and  sample  size  associated  with  the  experiment  resulted  in  data  and 
recommendations  with  reduced  reliability.  According  to  many  program  managers,  the 
experiment’s  results  were  also  influenced  by  politics.  Finally,  the  applied  results  of  the 
AWE  made  many  managers  question  the  utility  of  the  exercise.  Acquisition  managers 
participating  in  future  AWEs  must  take  active  measures  to  increase  the  likelihood  of 
achieving  benefits  for  their  programs. 

The  potential  benefits  to  acquisition  programs  participating  in  AWEs  include: 

•  marketing  and  exposure  of  program 

•  early  user  feedback 

•  refinement  of  user  requirements 

•  development  of  Tactics,  Techniques,  and  Prodedures  (TTP) 

•  follow-on  support  for  funding  and  production  decisions 

•  information  on  integration,  interfaces,  and  interoperability 

•  exposure  of  developers  to  the  user's  environment 
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Multiple  factors  contribute  to  a  program  manager’s  ability  to  benefit  from  AWE 
participation.  Program  managers  should  develop  objectives  and  measurement  processes 
for  the  AWE  that  will  identify  product  improvements  and  areas  for  future  investment. 
The  results  of  AWE  participation  should  be  adequately  documented  to  allow  the  use  of 
AW^  derived  information  and  findings  throughout  the  developmental  lifecycle  of  the 
participating  program.  To  achieve  maximum  benefits  from  the  AW^,  the  Decision  to 
Experiment  Ladder  should  be  implemented.  Acquisition  managers  should  actively 
participate  in  the  AWTE  planning  process  and  tailor  their  program’s  activities  in  the  AWE 
to  relate  to  the  issues  and  decisions  facing  the  program.  Data  collection  and  analysis 
plans  should  also  be  tailored  to  insure  that  the  information  derived  from  AW^ 
participation  is  of  value  to  the  program.  Finally,  program  maturity  has  a  impact  on  an 
initiative’s  ability  to  benefit.  While  programs  at  all  levels  of  maturity  can  benefit  from 
AWE  participation,  initiatives  in  mid-developmental  phases  of  acquisition  that  are 
sufficiently  mature  and  ruggedized  to  tolerate  the  harsh  environment  of  AWEs  are  best 
positioned  to  benefit.  Also,  these  programs  have  architectures  that  ^e  not  yet  finalized 
and  can  make  the  most  use  of  information  derived  from  participation. 

Program  managers  should  conduct  a  detailed  cost  and  benefit  analysis  to  decide  if 
AWE  participation  will  be  advantageous  to  their  program.  Costs  in  terms  of  protoyping, 
manning,  fielding,  transportation,  and  schedule  should  be  included.  Potential  benefits 
should  be  weighed  against  potential  risks  involved  with  AWE  participation.  These 
potential  risks  include  poor  return  on  investment,  potential  negative  exposure  for  the 
program,  and  excessive  changes  in  requirements  definition.  This  chapter  presented  a 
model  to  conduct  risk  analysis  for  AWE  participation  based  on  derived  risk  factors.  This 
risk  analysis  method  should  be  tailored  to  the  program’s  specific  situation  and  the 
program  manager’s  tolerance  for  risk.  The  risks  associated  with  AWE  participation  can 
be  mitigated  by  early  budgeting  of  funds  for  the  AWE,  marketing  to  the  user,  placing 
significant  emphasis  on  user  training,  and  by  directly  participating  in  the  AWE  planning 
process. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  AWE  Objectives  and  Structure 

The  stated  TF  XXI  AWE  objectives  included:  (1)  experimenting  with  advanced 
technologies  and  concepts  that  leverage  capabilities  of  information  age  technologies  and 
(2)  providing  information  to  support  investment  decisions  on  the  most  promising 
initiatives  [ECC,  1996].  The  AWE  was  also  meant  to  help  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC)  refine  requirements  and  develop  solutions  for  Force 
XXI.  The  AWE  Experiment  Directive  published  five  specific  objectives  for  the  AWE: 

a.  Assess  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2) 
Applique  and  Tactical  Internet  capabilities  to  refine  FBCB2  requirements. 

b.  Assess  digitized  brigade  combat  service  support  concept. 

c.  Experiment  with  advanced  technologies  and  concepts  that  leverage 
capabilities  of  information  age  technologies. 

d.  Refine  digitized  tactics,  techniques,  and  procedures  (TTP)  for 
brigade  operations. 

e.  Provide  information  to  support  investment  decisions  on  the  most 
promising  initiatives.  [ECC,  1996] 

The  Task  Force  XXI  Advanced  Warfighting  Experiment  was  not  designed  to 
provide  significant  benefits  to  the  acquisition  team  of  program  manager  and  user 
representative.  The  nature  of  the  AWE  experiment  limited  the  potential  derived  benefits 
for  the  program  manager.  The  acknowledged  AWE  limitations  in  repeatability,  data 
collection,  interaction  effects,  and  sample  size,  precluded  any  summative  findings 
However,  participating  initiatives  can  benefit  from  formative  findings  derived  from  the 
AWE  data  and  take  measures  to  increase  the  potential  to  benefit  from  the  experiment. 

2.  Use  of  Recommendations  to  Support  Investment  Decisions 

The  specific  recommendations  derived  from  the  Task  Force  XXI  Advanced 
Warfighting  Experiment  (AWE)  were  used  to  support  investment  decisions  and  to  refine 
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requirements  of  participating  initiatives  in  some  cases.  Overall,  programs  reported  that 
fifty-two  percent  of  the  OPTEC  recommendations  from  the  AWE  Live  Assessment 
Report  were  either  fully  or  mostly  implemented.  Thirty  percent  of  recommendations 
were  not  implemented  at  all.  Respondents  indicated  only  a  moderate  benefit  from 
participating  m  the  AWE  and  that  the  data  and  reconunendations  received  were  only 
somewhat  valuable.  The  most  cited  reason  for  recommendations  not  being  implemented 
was  a  lack  of  funding.  The  recommendations  were  most  likely  to  be  implemented  if  it 
was  a  high  priority  of  the  user. 

The  perceived  value  of  the  derived  AWE  data  was  not  a  significant  factor  in  the 
implementation  of  the  AWE  recommendations.  The  data  indicates  that  changes  to  a 
program  were  more  often  implemented  based  on  user  requirements  and  funding 
availability,  rather  then  on  OPTEC’s  recommendation.  Participants  considered  data 
valuable  in  making  decisions  as  acquisition  managers  when  the  met  one  of  three 
requirements.  First,  the  data  was  valuable  when  it  provided  actual  user  feedback  on 
specific  user  requirements.  Second,  data  was  considered  valuable  when  it  contributed  to 
the  development  of  tactics,  techniques,  and  procedures.  Finally,  the  data  was  considered 
valuable  when  it  was  provided  at  a  time  when  it  could  be  instrumental  in  refining 
requirements  and  design. 

3.  Contributing  Factors  to  a  Program’s  Ability  to  Benefit 

It  can  be  shown  from  the  collected  data  that  the  level  of  recommendation 
implementation  and  the  perceived  level  of  benefit  from  the  AWE  were  generally  related. 
It  cannot  be  concluded,  however,  that  the  implementation  of  AWE  recommendations  was 
solely  responsible  for  a  respondents  perception  of  benefit.  A  wide  range  of  confounding 
factors  effected  an  initiative’s  ability  to  benefit  from  the  AWE. 

a.  Definition  of  Benefit 

In  determining  the  contributing  factors  to  a  program’s  ability  to  benefit 
from  AWE  participation,  a  program  manager  must  first  define  the  term  benefit,  as  it 
relates  to  his  program.  Program  managers  reported  the  following  potential  benefits  froth 
their  TF  XXI  AWE  participation: 

•  marketing  and  exposure  of  program 

•  early  user  feedback 
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•  refinement  of  user  requirements 

•  developmentofTactics,  Techniques,  and  Prodedures(TTP) 

•  follow-on  support  for  funding  and  production  decisions 

•  information  on  integration,  interfaces,  and  interoperability 

•  exposure  of  developers  to  the  user’s  environment 

b.  Objective  Determination,  Measurement,  and  Documentation 

The  level  at  which  AWE  results  were  documented  within  the  acquisition 
management  office  had  a  direct  effect  on  the  ability  of  the  program  to  benefit  from 
participation.  A  subjective  analysis  of  survey  responses  provided  evidence  that  those 
program  managers  who  could  provide  a  detailed  explanation  of  the  specific 
developmental  issues  being  addressed  on  their  program  at  the  time  of  its  participation  in 
the  AWE  and  the  decisions  to  be  made  from  gathered  data,  generally  reported  a  much 
higher  level  of  perceived  benefit  from  the  experiment.  Those  programs  that  specifically 
developed  test  objectives  and  measurement  processes  linked  to  specific  acquisition 
decisions  to  be  made,  were  better  positioned  to  benefit  from  AWE  participation. 

c.  Tailoring  of  A  WE  Participation 

The  extent  a  manager  was  able  to  tailor  or  influence  a  program’s  specific 
activities  in  the  AWE  to  relate  to  the  program’s  acquisition  issues  and  decisions  directly 
contributed  to  the  extent  that  the  program  benefited  from  participating.  Those  initiatives 
reporting  a  high  level  of  ability  to  tailor  or  influence  generally  reported  much  higher 
levels  of  perceived  benefit  from  participation  in  the  AWE. 

d.  Program  Maturity 

Program  maturity  has  a  impact  on  an  initiative’s  ability  to  benefit.  While 
programs  at  all  levels  of  maturity  can  benefit  from  AWE  participation,  initiatives  in 
PDRR  or  EMD  that  are  sufficiently  mature  and  niggedized  to  tolerate  the  harsh 
environment  of  AWEs  are  best  positioned  to  benefit.  Also,  these  programs  have 
architectures  that  are  not  yet  finalized  and  can  make  the  most  use  of  information  derived 
from  participation. 
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Risk  Assessment 


e. 

Survey  comments  indicate  a  substantial  risk  to  programs  participating  in 
an  AWE.  Program  risks  include  factors  beyond  the  costs  of  participation,  to  include  a 
poor  return  on  investment,  potential  negative  exposure,  and  extensive  changes  in 
requirements.  The  factors  contributing  to  risk  include: 

•  Maturity 

•  Ruggedization  and  maintainability 

•  The  focus  of  the  AWE 

•  Funding  availability 

•  Equipment  availability 

•  Biases 

•  Status  of  production  decisions 

4.  Characteristics  of  Programs  Best  Positioned  to  Benefit  from  AWE 
Participation 

Based  on  the  data  presented  in  Chapters  IV  and  V,  a  program’s  potential  to  gain 
valued  investment  and  requirements  information  from  participation  in  Advanced 
Warfighting  Experiments  can  be  evaluated  based  on  the  following  factors: 

•  State  of  the  technology  and  how  the  initiative  relates  to  the  goals  of  the  AWE 

•  The  ability  to  tailor  activities  to  relate  to  the  needs  of  the  program 

•  Development  of  clear  objectives,  expected  outcomes,  and  strategies  for 
implementation 

•  System  maturity 

•  Ruggedization  and  maintainability 

•  Equipment  availability 

•  Funding  availability 

•  Data  requirements 

•  Risk  situation 

a.  State  of  the  Technology  and  Goals  of  the  AWE 

Programs  positioned  to  receive  the  most  benefit  from  AWE  participation 
will  fit  within  the  published  goals,  objectives,  and  focus  of  the  AWE  as  stipulated  by  the 
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planning  officials.  Also,  the  AWE  must  address  the  issues  facing  the  initiative.  As 
demonstrated  in  the  TF  XXI  AWE,  those  programs  that  are  not  a  high  priority  of  the 
analyzing  agency  may  not  receive  adequate  feedback.  Additionally,  initiatives  that  are 
new  concept  technologies  with  no  current  tactics,  techniques,  and  procedures  are  best 
positioned  to  benefit  from  the  integration  of  multiple  systems  in  an  experimental 
environment.  Finally,  programs  with  positive  preconceived  biases  and  general 
acceptance  by  leadership,  users,  and  evaluators  are  best  positioned  to  benefit.  Program 
managers  firom  the  TF  XXI  AWE  also  stated  that  incoming  biases  can  prevent  any  real 
analysis  of  a  system’s  performance  at  the  AWE. 

b.  The  Ability  to  Tailor 

The  extent  a  manager  was  able  to  tailor  or  influence  a  program’s  specific 
activities  in  the  AWE  to  reflect  the  program’s  current  acquisition  issues  and  decisions  had 
a  direct  impact  on  the  extent  to  which  the  program  benefited  firom  participating.  Those 
initiatives  reporting  a  high  level  of  ability  to  tailor  or  influence  generally  reported  much 
higher  levels  of  perceived  benefit  firom  participation  in  the  AWE.  Both  program 
managers  and  user  representatives  agree  that  to  achieve  maximum  benefit  from  AWEs, 
acquisition  managers  must  be  able  to  participate  in  the  planning  process. 

c.  Development  of  clear  objectives,  outcomes,  and  strategies 

The  data  suggests  that  those  program  managers  who  develop  detailed 
experiment  objectives  and  expected  outcomes,  systems  for  data  documentation  and 
analysis,  and  strategies  for  implementation  of  AWE  data  and  recommendations  will 
receive  more  valued  data  and  will  experience  more  benefit  from  AWE  participation. 

d.  Maturity 

Maturity  effects  the  ability  of  a  program  to  perform  adequately  as  well  as 
the  ability  of  the  initiative  to  implement  recommendations  derived  from  the  AWE. 
Programs  in  the  mid-developmental  phases  of  acquisition  are  best  positioned  to  benefit 
from  AWE  participation.  Programs  in  early  in  development  are  high  risk  because  of  their 
unpredictability  and  vaguely  defined  roles  and  requirements.  Programs  late  in 
development  or  in  production  are  medium  risk  because  they  may  be  to  far  in  development 
to  capitalize  on  recommendations. 
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Additionally,  programs  with  no  pending  production  or  funding  decisions 
are  best  positioned  to  benefit  from  AWE  participation.  Programs  that  have  secured 
approval  and  funding  for  production  prior  to  participation  in  an  AWE  but  have  not  yet 
begun  production,  face  the  added  risk  of  poor  performance  and  loss  of  support.  Those 
initiatives  with  imminent  production  decisions  are  medium  risk,  in  that  AWE 
performance  can  significantly  influence  the  survivability  of  the  system. 

e.  Ruggedization  and  Maintainability 

Systems  participating  in  AW^s  should  be  ruggedized  and  easily 
maintainable.  A  system  that  is  highly  rugged  is  able  to  withstand  the  stresses  associated 
with  operational  use  in  harsh  environments.  A  system  that  scores  low  in  ruggedization 
has  sensitive  components  with  maintenance  procedures  that  are  difficult  to  conduct  in  a 
field  environment.  Low  ruggedization  may  be  associated  with  early  prototype  systems. 

f.  Equipment  Availability 

Mature  initiatives  with  readily  available  production  items  are  best 
positioned  for  AWE  participation.  Development  of  prototypes  or  surrogates  requires 
additional  costs,  time,  and  developmental  testing. 

g.  Funding  Availability 

AWE  initiatives  must  have  sufficient  funding  to  benefit  from  AWHE 
participation.  Participation  in  an  AWE  is  resource  intensive.  A  program  must  have 
sufficient  funds  for  prototypes,  manning,  training,  transportation,  and  maintenance. 
Required  funding  will  vary  based  on  the  complexity  of  the  system.  Additional  funds  will 
not  normally  be  available  specifically  for  AWTE  participation.  Sufficiently  funded 
programs  will  have  budgeted  dollars  specifically  for  the  AWTE.  Insufficiently  funded 
programs  will  have  tight  budgetary  constraints 

h.  Data  Requirements 

Programs  with  extensive  need  for  data  on  integration,  interfaces, 
interoperability,  and  user  requirements  are  best  positioned  to  benefit.  Those  programs 
early  in  Concept  Exploration  would  score  have  the  added  benefit  of  receiving  early  user 
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feedback  to  refine  system  requirements.  Those  programs  with  sufficient  feedback  from 
other  sources,  or  are  not  in  a  position  to  implement  any  recommendations  from  the 
feedback  cannot  take  significant  advantage  of  AWE  results.  Also,  those  programs  with 
manager’s  and  engineers  having  little  exposure  to  the  user’s  environment,  are  positioned 
to  gain  valued  experience. 

L  Risk  Situation 

AWE  participation  is  not  without  risk.  Programs  with  a  high  tolerance  for 
risk  are  best  positioned  to  participate  in  AWES.  Acquisition  programs  may  be  negatively 
effected  by  AWE  participation.  Participating  programs  will  experience  significant  costs 
associated  with  the  AWE  that  may  not  provide  a  return  on  investment.  AWE  initiatives 
might  receive  negative  exposure,  effecting  future  funding  and  development  decisions. 
Finally,  AWEs  can  reveal  drastic  changes  in  requirements  that  a  program  might  not  be 
positioned  to  tolerate. 

B.  RECOMMENDATIONS 

1.  Acquisition  Managers 

a.  Cost  Benefit  Analysis 

Acquisition  managers  should  conduct  a  detailed  cost  and  benefit  analysis 
to  decide  if  AWE  participation  will  be  advantageous  to  their  program.  Costs  in  terms  of 
protoyping,  manning,  fielding,  transportation,  and  schedule  should  be  included. 

b.  Formulation  of  Objectives  and  Measurements 

To  maximize  the  potential  to  benefit  from  AWE  participation,  acquisition 
managers  should  develop  objectives  and  measurement  processes  for  the  AWE  that  will 
identify  product  improvements  and  areas  for  future  investment.  The  results  of  AWE 
participation  should  be  adequately  documented  to  allow  the  use  of  AWE  derived 
information  and  findings  throughout  the  developmental  lifecycle  of  the  participating 
program. 


99 


c.  Participation  in  the  Planning  Process 

To  achieve  maximum  benefits  from  the  AWE,  acquisition  managers 
should  actively  participate  in  the  AWE  planning  process  and  tailor  their  program’s 
activities  in  the  AWE  to  relate  to  the  issues  and  decisions  facing  the  program.  Data 
collection  and  analysis  plans  should  also  be  tailored  to  insure  that  the  information  derived 
from  AWE  participation  is  of  value  to  the  program. 

d.  Risk  and  Benefit  Comparison  and  Analysis 

Acquisition  managers  contemplating  participation  in  future  AWEs  should 
conduct  a  detailed  analysis  of  the  risks  associated  with  AWEs  and  consider  their  tolerance 
for  risk  as  a  factor  in  making  a  decision  to  participate.  Potential  risks  should  be 
compared  to  potential  benefits  from  participation  and  interpreted  to  determine  the  best 
course  of  action  for  the  program.  Active  measures  should  be  identified  to  mitigate  the 
specific  risks  associated  with  AWE  participation.  Methods  to  mitigate  risk  include  (1) 
early  budgeting  of  funds  for  the  AWE,  (2)  marketing  to  the  user,  (3)  placing  significant 
emphasis  on  user  training,  and  (4)  direct  participation  in  the  AWE  plaiming  process. 

2.  AWE  Planners 

a.  Acquisition  Manager  Involvement 

For  the  JCF  AWE,  OPTEC  representatives  will  meet  with  program 
managers  to  determine  issues  that  are  linked  to  AWE  objectives  [TRADOC  JCF-AWE, 
1998].  Planners  for  future  AWEs  should  allow  acquisition  managers  to  actively 
participate  in  the  development  of  AWE  goals  and  objectives,  scenarios,  and  data 
collection  and  analysis  plans  so  that  the  information  derived  from  AWE  participation  is 
of  value  to  the  program. 

b.  AWE  Funding 

The  Army  should  consider  providing  funding  for  AWE  initiatives  so  that 
acquisition  managers  can  increase  the  benefits  derived  from  participation.  No  additional 
funding  will  normally  be  provided  to  the  program  manager  for  fumre  AWE  participation 
or  for  implementation  of  any  derived  recommendations.  Participation  in  AWEs  requires 
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expensive  prototyping,  manning,  fielding,  training,  and  transportation  costs  that  must  be 
drawn  from  existing  research  and  development  accounts.  Program  managers  cannot 
increase  their  roles  in  the  AWE  process  and  ability  to  tailor  activities  without  dedicated 
support  from  the  Army’s  budgetary  process. 

C.  SUGGESTED  FURTHER  STUDY 

1.  Cost  Benefit  Analysis 

Conduct  a  detailed  cost  benefit  or  cost  effectiveness  analysis  of  AWE 
participation.  Include  costs  associated  with  prototyping,  training,  fielding,  manning, 
transporting,  and  supporting  participation  in  the  experiment  as  compared  to  the  benefits 
received.  This  study  will  assist  the  program  manager  in  making  participation  decisions 
based  on  quantifiable  data. 

2.  A  Study  of  Increased  Acquisition  Manager  Participation  in  the 
Planning  Process 

Conduct  a  study  of  most  recently  completed  as  well  as  future  AWEs  where 
planners  actively  promote  the  participation  of  acquisition  managers.  Determine  if 
increased  benefit  is  derived  from  this  participation  with  or  without  an  additional 
commitment  of  funding  specifically  for  AWE  participation. 

3.  Risk  Analysis  Study 

Conduct  a  detailed  study  of  all  risks  associated  from  AWE  participation  and 
validate  a  formal  risk  analysis  and  mitigation  evaluation  system  through  use  on  acmal 
systems  participating  in  AWEs.  This  validated  risk  analysis  and  mitigation  plan  will 
assist  acquisition  managers  in  deriving  the  higher  levels  of  benefit  from  AWE 
participation  while  being  exposed  to  less  risk  of  a  poor  return  on  investment,  negative 
exposure,  or  extensive  changes  in  requirements. 
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4.  Study  of  Contributing  Factors  to  Beneflting  from  Current  AWEs. 

Conduct  a  detailed  study,  similar  to  this  thesis,  on  more  current  AWEs  to  include 
the  Division  XXI  AWE  and  the  Joint  Contingency  Force  (JCF)  AWE  to  determine  if  the 
findings  and  recommendations  from  this  thesis  are  applicable  to  other  AWEs. 
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APPENDIX  A.  SPECIFIC  INITIATTVE  RECOMMENDATIONS  FROM  THE  TF 
XXI  AWE  LIVE  EXPERIMENT  ASSESSMENT  REPORT 


Tactical  Unmanned  Aerial  Vehicle  (TUAV) 

-  A  target  location/designation  capability  should  be  included  in  the  objective  TUAV 
that  allows  targeting  operations  from  a  standoff  position. 

-  Antennas  on  the  TUAV  should  be  positioned  so  that  communications  with  ground 
terminals  can  be  maintained  during  all  aspects  of  flight. 

-  Users  indicated  that  it  was  important  for  the  TUAV  to  be  able  to  support  imagery 
exploitation  at  the  target  identification  level.  It  must  be  able  to  distinguish 
between  weapon  model  types  (I.e.,  M-1  versus  1-12). 

-  Every  TOC  should  have  the  ability  to  receive  a  direct  feed  from  a  UAV  operating 
in  its  area. 

-  Video  display  needs  to  include  telemetry  data  so  users  know  when  the  UAV  is  in 
its  area  (telemetry  data  was  not  relayed  to  battalion  TOCs  via  RVT  or 

-  Users  indicated  that  they  needed  more  tools  for  digitizing  imagery  (e.g.,  an 
imagery  editing  station  that  would  permit  them  to  grab  a  still  frame  and  annotate  it 
prior  to  dissenoination). 

-  Users  reported  during  interviews  that  the  UAV  downlink  needs  to  be  digital  and 
encrypted.  They  were  concerned  that  the  enemy  could  intercept  the  down-link 
and  benefit  from  its  output. 

Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  (Applique) 

-  Continue  experimentation  with  Applique/FBCB2  using  other  interface  devices. 
Soldiers  were  not  satisfied  with  the  keyboard/mouse  configurations,  especially 
during  movement.  Recommend  experimentation  with  more  function  key,  touch 
screen,  or  light  pen  systems  as  an  interim  to  voice  activated,  hands-free  system. 

-  Determine  the  most  critical/useful  functions  and  eliminate  non-critical 

-  Improve  vehicle  hardware  integration. 

-  During  future  experimentation/testing  with  the  Applique,  perform  technical  testing 
to  ensure  that  specifications  for  environmental  and  EMl/^C/NBC  hardening  are 
met.  A  particular  area  of  concern  is  the  possible  EMI  effects  within  designated 
and  planned  combat  platforms. 
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-  Ensure  that  prototypes  of  the  acquisition  version  of  the  DSSU  are  subject  to  as 
many  experiments/user  juries  as  possible,  in  order  to  further  refine  and  define 
operating  systems  and  intended  capabilities. 

-  Relook  the  DSSU  distribution  plan,  based  upon  the  refinement  of  intended 
capabilities.  Certain  functions  may  not  be  necessary  at  lower  levels  of  command 
and  control. 

-  Continue  to  develop  and  mature  the  Applique  CSS  functions.  While  the  Applique 
did  not  perform  well  during  the  experiments,  it  has  great  potential  in  the  CSS 

Long-Range  Advanced  Scout  Surveillance  System  (LRAS3)  and  the  Hunter 

-  Develop  the  LRAS3  or  HS3  for  use  by  the  Brigade  Reconnaissance  Troop  and 
scout  organizations. 

-  Incorporate  system  survivability  improvements  into  development  plan. 

-  Several  comments  noted  that  at  times  the  HS3  system  was  hard  to  emplace  and 
the  stowed  mast  can  make  the  vehicle  unbalanced. 

-  There  were  concerns  that  the  HS3  system  gave  off  too  much  of  a  signature, 
could  be  easily  spotted  by  enemy  soldiers  and  did  not  have  a  crew  served  weapon 
with  the  vehicle  to  help  make  the  crew  and  vehicle  more  survivable.  It  was 
suggested  that  the  system  be  put  in  a  vehicle,  such  as  an  Ml  13,  that  would  allow 
the  system  to  be  stowed  under  armor  protection,  give  the  crew  a  better  work  area 
and  provide  better  cross  country  travel  capability. 

-  The  capability  to  dismount  the  LRAS3  system,  place  the  system  on  an 
observation  post,  and  hide  the  vehicle  would  improve  system  and  BRT 

Javelin  Antitank  Weapon  System 

-  Improve  day  sight  to  enable  the  gunner  to  identify  out  to  the  range  of  the  weapon. 

-  Improve  the  battery  life  of  the  CLU  from  current  specification  of  4  hours  to 
12-24  hours. 

-  Adjust  the  communication  system  distribution  plan  to  allow  for  proper 
communications  of  AT  section  personnel. 

Lightweight  Video  Reconnaissance  System  (LVRS) 

-  Increase  conununication  range  of  the  LVRS  to  25-30  KMs. 

-  When  the  LVRS  initiative  is  more  mature  (improved  reliability  and 
communications  capability),  that  it  be  included  in  a  future  experiment. 
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-  Modify  the  LVRS  so  that  scouts  can  operate  remotely  and  emplace  the  camera,  in 
order  to  move  to  a  covered  and  concealed  location  to  monitor  the  screen. 

Target  Location  and  Observation  System  (TLOS) 

-  Investigate  the  concern  expressed  by  some  users  indicating  the  laser  emitted  by 
the  TLOS  would  compromise  their  position. 

-  Consider  use  of  the  initiative  in  future  exercises  when  the  initiative  has  matured 
(improved  reliability). 

Mortar  Fire  Control  System  (MFCS) 

-  Recommend  that  an  experiment  or  test  be  set  up  so  that  data  on  the  fire  mission 
processing  time,  at  each  of  the  fire  mission  processing  nodes,  can  be  collected  so 
an  evaluation  of  MFCS  affects  on  timeliness  of  mortar  fires  can  be  completed. 

-  Improve  communications  between  the  MFCS  and  the  AFATDS  at  the  BN  FSE. 

Stingray  Combat  Protection  System 

-  Continue  system  development  with  emphasis  on  increasing  system  reliability. 

-  Investigate  if  the  maximum  effective  range  requirements  are  adequate  to  best 
support  the  force. 

Light  Tactical  Operations  Center  (LTOC) 

-  Continue  experimentation  and  development  of  the  LTOC  as  digital  systems  and 
initiatives  are  improved. 

-  Place  emphasis  on  digital  systems’  hardware  and  software  integration 
development  to  improve  the  automatic  sharing  and  integration  of  information 
among  the  systems  in  the  LTOC  and  improve  staff  mission  planning  and 

-  In  addition  to  preformatted  messages,  develop  common,  user  friendly,  e-mail 
message  type  capability  for  all  digital  information  passing  systems. 

-  Develop  tactics,  techniques,  and  procedures  to  cover  the  operation  of  the  LTOC. 

OH-58D  Kiowa  Warrior 

Increase  the  maneuver  unit  commander  and  staff  training  on  the  use  and  utility  of 
the  near  real  time  picture  of  the  battlefield  provided  by  the  OH-58D. 

-  Provide  the  capability  to  receive  the  near  real  time  picture  at  all  maneuver  battalion 
TOC,  not  just  the  AVTOC. 
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-  Fix  the  problems  with  the  image  file  format  and  the  baud  rate  of  the  modem  used 
to  transfer  the  images. 

-  Consider  a  video  down  link  similar  to  the  UAV. 

-  Fix  problems  with  SINCGARS  SIP  radio,  line  of  sight  reception  and  reliability  in 
system. 

Army  Airborne  Command  and  Control  System  (A2C2S) 

-  Increase  maneuver  unit  commander  and  staff  training  on  the  use  and  utility  of  the 
real-time  picture  of  the  battlefield  provided  by  the  A2C2S. 

-  Relook  the  crash  survivability  of  the  A2C2S  physical  layout. 

-  Provide  additional  ground  and  air  radios  with  increased  range  and  data  handling 
capability  to  correct  the  identified  limitations  of  the  SINCGARS  SIP  and  power 
output  of  the  FH  MUX. 

-  Fix  the  file  transfer  problems  prior  to  production  of  the  A2C2S. 

-  The  A2C2S  should  have  fewer  MCS  workstations  and  should  have  AS  AS  and 
AFATDS,  instead.  The  A2C2S  should  have  at  least  five  radio  nets. 

-  Some  human  factors  engineering  improvements  are  needed,  such  as  lighting, 
available  workspace,  and  intercom  switch  placement. 

Aviation  Tactical  Operations  Center  (AVTOC) 

-  Improve  mobility  of  the  AVTOC.  It  took  approximately  3-4  hours  to  tear  down 
and  six  hours  to  set  up  using  all  of  the  TOC  personnel. 

-  Provide  training  for  commanders  and  staff  personnel  on  ATCCS  in  terms  of 
capabilities,  functionality,  and  management  of  assets. 

-  Allow  for  integration  of  additional  liaison  officers  and  their  equipment  which  were 
not  part  of  the  AVTOC  system,  e.g.,  field  artillery  and  USAF  UNO’s. 

-  Improve  communications  and  connectivity  between  the  ATCCS,  MCS,  Applique, 
and  AMPS. 

-  Provide  additional  MCS^  ASAS,  and  AFATDS  systems  to  allow  for  future 
planning  while  conducting  current  operations. 

-  Provide  a  wireless  LAN  for  workstations  to  reduce  setup/tear-down  times  and 
clutter  within  the  TOC. 

AH-64D  Longbow  Apache  (LB A) 

-  Increase  maneuver  unit  commander  and  staff  training  on  the  use  and  utility  of  the 
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near  real-time  picture  of  the  battlefield  provided  by  the  LBA. 

-  Provide  the  capability  to  receive  near  real-time  picture  at  all  maneuver  battalion 
TOC,  not  just  the  AVTOC,  Maneuver  Bde  TOC,  and  A2C2S. 

-  Solve  the  interface  problem  between  the  IDM  and  EPLRS. 

-  Solve  the  problem  of  being  able  to  select  the  priority  of  JSTARS  or  LBA  FCR 
data  feeds  into  the  CGS(P). 

-  Make  the  AVTOC  target  type  data  field  match  the  LBA  target  type  data  field  to 
enable  the  AVTOC  to  recognize  the  target  type  from  the  LBA  shot  at  files. 

-  Fix  the  problem  with  the  A2C2S  IDM  sending  LBA  FCR  data  to  its  file  server  to 
enable  the  A2C2S  to  display  the  LBA  FCR  data  on  a  map  overlay  to  all  its  work 
stations  and  the  large  screen  display. 

-  Provide  a  means  of  disseminating  the  LBA  targets  data  files  to  multiple  users, 
such  as  posting  of  enemy  SA  on  Applique  after  AS  AS  review. 

Aviation  Mission  Planning  System  (AMPS) 

-  Make  AMPS  fully  interoperable  with  MCS  for  connectivity  or  combine  the 
capabilities  of  both  systems  into  one  unit. 

-  Replace  dot  matrix  printer  with  a  faster  more  capable  printer  with  laser  print 

-  Increase  AMPS  screen  size  to  allow  collective  use  and  mission  briefings. 

-  Make  software  more  user  friendly  with  Microsoft  Office  type  work  processing 
capability. 

-  Make  the  import  of  situational  data  simpler.  AMPS  did  allow  the  import  of 
situational  data  however,  that  data  had  to  be  manually  loaded  at  the  AMPS 
workstation  before  it  could  be  passed  to  another  AMPS.  Generally,  it  was  too 
difficult,  not  enough  time  was  available  to  perform  the  required  tasks. 

Advanced  Field  ArtiUery  Tactical  Data  System  (AFATDS) 

-  Add  target  strength  to  the  software  logic  protocol  consideration  queue  for  the 
calculus  of  determining  a  target  value. 

-  Modify  the  software  functionality  for  counter-fire  to  appropriately  reflect  Army 
doctrine  on  counter-fire  missions. 

-  Modify  the  AFATDS/LCU  software  to  provide  the  FSCOORD  with  current  fire 
mission  data. 

-  Improve  some  user  interface  capabilities.  Modifying  the  Attack  Methods  Table 
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information  must  be  easier  for  the  user  to  do  during  the  course  of  the  battle. 

-  Improve  some  user  interface  capabilities.  The  battalion  FDO  should  have  the 
capability  to  modify  a  mass  fire  order  at  his  OPFAC  when  needed,  and  be  able  to 
send  that  modified  fire  order  to  multiple  subordinates  simultaneously.  This  would 
allow  him  to  retain  digital  interface  with  subordinated  OPFACs  while  tailoring  the 
tactical  fire  solution  to  current  battlefield  realities. 

-  Develop  a  better  communications  system  than  SINCGARS-SP  currently  provides 
for  AFATDS  OPFACs. 

Experimental-Fire  Support  Team  (X-FIST) 

-  Improve  the  maintenance  concept  of  the  X-FIST  targeting  station.  The  targeting 
station  maintenance  personnel  should  deploy  for  combat  with  the  maneuver  unit. 

-  Review  the  LCU  hard  drive  and  RAM  issue  to  determine  if  more  RAM  is 
necessary  or  if  a  hardware  modification  might  allow  the  user  to  operate  the 
system  successfully  during  operations  on  the  move.  When  on  the  move  the  LCU 
hard  drive  tended  to  crash  requiring  the  system  to  be  shut  down  while  on  the 

Paladin/Field  Artillery  Ammunition  Support  Vehicle  (FAASV) 

-  The  Paladin/FAASV  was  identified  as  a  "high-performing"  system  and  fielding 
should  be  continued. 

Fire  Direction  Center  Vehicle  (FDC-V) 

-  The  FDC-V  should  continue  as  a  viable  candidate  to  replace  the  1068  as  the 
FDC/POC  operations  vehicle. 

Lightweight  Laser  Designator  &  Rangefinder-Surrogate  (LLDR-Surrogate) 

-  Field  the  initiative  as  a  replacement  for  GAT.LD  in  scout  and  Striker  forces. 

-  Provide  a  "slave"  capability  for  the  LLDR  from  a  vehicle  that  will  provide  longer 
use  capability  for  the  observers. 

Linebacker  and  Avenger 

-  The  Avenger  and  Linebacker  were  assessed  as  "high-performing"  initiatives 
during  the  conduct  of  the  experiment.  Develop  tactics,  techniques,  and 
procedures  that  fully  support  the  new  Linebacker/Avenger  system  improvements. 

Forward  Area  Air  Defense  Command,  Control  and  Intelligence  (FAAD  C2I) 

-  FAAD  C2I  was  identified  as  a  "high-performing"  initiative  during  the  conduct  of 
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the  TF  XXI  AWE.  Develop  TTPs  that  fully  support  the  new  FAAD  C21/Sentinel 
system  improvements. 

-  WTiile  operating  in  the  "HOVER"  mode  to  detect  low-flying  RW  aircraft,  the 
increased  radar  sensitivity  resulted  in  clutter  on  the  unit  display.  In  addition  to 
helicopters,  ground  clutter  and  vehicles  appeared  on  the  EO  and  SHTU  display. 

Wide  Area  Munitions  (WAM)  Hornet 

-  The  self-arming  battery  should  be  increased  to  six  hours  versus  the  current 
two-hour  life  once  the  WAM  Hornet  is  installed. 

-  The  remote  arming  range  needs  to  be  increased  from  lOOOm  to  3000m. 

-  The  training  device  for  the  XM97  needs  to  be  redesigned  to  incorporate  remote 
arming  and  the  remote  arming  capability  needs  to  incorporate  visual  sign  allowing 
observation  that  the  WAM  Hornet  is  armed  or  recycled. 

-  The  commander  needs  to  have  a  flexible  employment  strategy  that  allows  him  to 
employ  the  WAM  Hornet  to  mass  the  effects  of  the  Hornets  and  to  ensure  that  the 
proper  integration  of  fires  is  accomplished.  The  X  pattern  and  Gauntlet  are  still 
good  employment  strategies,  but  hornet  employment  shouldn’t  be  limited  to  just  X 
and  Gauntlet. 

Automated  Nuclear,  Biological,  and  Chemical  Information  System  (ANBACIS) 

-  rr&E  IPT  should  include  an  operational  test  as  part  of  the  normal  acquisition 
strategy  for  the  ANBACIS . 

-  Establish  seamless  connectivity  between  Applique  and  MCS/P. 

Radio  Frequency  Tags/Identification  Devices  (RF  Tags),  PLS-E  Enhancements 

-  Develop  a  mobile  interrogator.  An  interrogator  mounted  on  a  trailer  with  a  cellular 
phone  could  be  easily  transported  from  site  to  site. 

-  Continue  to  pursue  RF  technology  to  include  improvement  of  components  and 
interservice  interoperability. 

-  Attention  must  be  given  to  development  of  control  procedures  for  use  of  AIT 
technology  in  the  field. 

-  Integrate  the  equipment  required  to  install  and  support  network  operations  into  the 
appropriate  supply,  transportation  and  ammunition  automated  information 
systems.  Supporting  these  systems  with  RF  technology  must  be  "fly-away" 
packages  capable  of  quick  installation  and  easier  maintenance. 
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Personnel  Service  Support  Control  System  CPSSCS) 

-  Fully  integrate  the  Force  XXI  Manning  System  (FMS)  within  CSSCS  so  that 
accurate  and  near-real  time  strength  reporting  can  occur.  Presently,  FMS  is  a 
prototype  of  the  desired  personnel  functionality  that  is  needed  in  CSSCS. 

-  Include  an  additional  table  or  data  black  that  shows  the  task  force  that  the  soldier 
is  attached  to  in  addition  to  his/her  UIC  that  is  already  listed. 

-  Continue  the  development  of  FMS  to  make  it  more  user  friendly.  Commonly 
used  custom  queries  should  be  added  to  the  reports  section  within  FMS. 

-  Provide  an  automated  interface  between  Applique  and  FMS.  The  manual  data  that 
had  to  be  inputted  and  managed  detracted  from  the  potential  responsiveness  of 
the  system. 

Medical  Communications  for  Combat  Casualty  Care  (MC4) 

-  Test  Telementoring,  Teleconsultation,  and  the  MDA  in  a  test  event  with  less 
OPTEMPO  than  the  TF  XXI  AWE  where  their  capabilities  can  be  fully 
assessed/evaluated. 

-  Perform  appropriate  studies  which  address  concerns  on  the 
survivability/vulnerability  of  AMEV  as  a  forward  area  evacuation  platform  I.e., 
replacement  for  the  Ml  13. 

-  Reevaluate  doctrine  for  Telementoring  and  Teleconsultation  to  determine  how  far 
forward  they  can  be  effective  on  the  battlefield. 

-  Improve  the  users  ability  to  load  litters  into  the  AMEV  and  securing  them  into  the 
top  litter  rack. 

Command  &  Control  Vehicle  (C2V)  and  Battle  Command  Vehicle  (BCV) 

-  Continue  procurement  process  on  C2V. 

-  No  further  development  should  be  considered  on  the  BCV  until  unacceptable 
communications  problems  are  resolved. 

-  Perform  a  detailed  technical  analysis  on  the  effect  on  conmiunications  of  the  large 
number  of  antennas  and  signal  emitters  in  the  Bn/Bde  TOC  areas. 

-  Test  the  C2V,  BCV,  and  digital  TOC  in  a  long-distance  Command  Post  Exercise 
(CPX)  environment  with  extensive  operations  on  the  move  in  order  to  fully 
demonstrate  their  utility  in  operations  on  the  move. 

-  Investigate  and  resolve  the  C2V  antenna  system.  The  system  poses  a  hazard  to 
soldiers,  xmder  certain  circumstances. 
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-  Numerous  suggestions  were  made  that,  to  reduce  motion  sickness  in  the  C2V, 
crew  stations  needed  to  face  toward  the  front  of  the  vehicle  at  all  times. 

Situational  Awareness  Data  Link  (SADL)  and  Battlefield  Combat 

-  Fully  investigate  and  resolve  the  reasons  for  the  incomplete  passing  of  digital  9-line 
reports. 

-  Develop  a  reliable,  rugged  display  unit  for  use  on  an  active  battlefield. 

-  Continue  development  of  the  S  ADL-EFAC  system. 

Battlefield  Combat  Identification  System  (BCIS) 

-  Fully  integrate  all  identification  systems  with  the  weapon  firing  system,  similar  to 
the  Ml  version  of  BCIS,  so  that  the  gunner  can  make  an  engagement  decision 
without  breaking  the  sight  picture. 

-  Provide  a  functional  system  for  dismounted  soldiers  to  prevent  them  from 
becoming  targets  or  engaging  friendly  vehicles. 

-  Perform  a  smaller  experiment  with  fully  equipped  units  to  render  more 
information  on  the  system. 

-  Investigate  whether  the  width  of  the  BCIS  interrogation  beam  prevents  positive 
ID  on  a  battlefield  with  friendly  and  enemy  vehicles  intermingled. 

Asynchronous  Transfer  Mode  (ATM) 

-  Examine  the  use  of  high-speed  multiplexer  cards  to  support  VTC  data  feed  as  an 
alternative  to  the  current  ATM. 

-  Implement  ATM  with  higher  data  rates,  more  bandwidth,  dynamic  bandwidth 
allocation,  and  high-speed  multiplexing  to  reduce  load  on  the  MSE  system. 

-  Simplify  the  difficult  methods  to  install,  operate,  and  maintain  the  ATM  switches. 

Surrogate  Digital  Radio  (SDR) 

-  The  SDR  was  identified  as  a  "high-performing"  initiative  during  the  conduct  of  the 
TF  XXI  AWE.  Continue  development  of  TTPs  which  will  ensure  that  it  is 
utilized  properly  in  the  future. 

-  Increase  bandwidth  and  throughput  in  the  fielded  version  of  SDR. 

Global  Broadcast  Service/Battlefield  Awareness  Data  Dissemination 

-  Develop  methods  for  users  to  properly  manage  the  increased  information 
provided  by  the  GBS/BADD. 
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-  Continue  further  development  of  the  GBS/B  ADD  and  associated  TTPs. 

-  Review  the  user  survey  results  and  incorporate  desired  features  into  the 
GBS/BADD  such  as  UAV  imagery,  LBA  messages,  weather  data,  and  commercial 

Air  Defense  Applique 

-  Improve  the  real  time  battlefield  situational  awareness,  digitized  command 
functions  and  connectivity  with  the  FAAD  C2I  network. 

-  Relocate  the  Applique  workstations  in  vehicles  to  ensure  ease  of  operation. 

-  Air  Defense  Liaison  officers  should  have  Applique.  They  currently  track  the 
battle  using  a  map  and  information  they  obtain  from  monitoring  the  voice  nets. 

-  Ammunition  re-supply  vehicles  should  be  equipped  with  Applique  to  assist  in 
timely  re-supply  operations. 

-  Include  an  automatically  scrolling  map  that  is  oriented  to  the  vehicle  direction  of 
travel  with  the  vehicle  icon  centered  on  the  display. 

-  Improve  the  Applique  menu  selection  option  procedures  by  making  them  more 
user  friendly. 

-  Applique  digital  network  is  most  effective  when  the  unit  is  stationary.  This  is  due 
to  the  difficulty  of  using  the  mouse  and  keyboard  on  the  move.  Develop  an 
improved  input  device  for  the  Applique. 

-  Improve  the  resolution  of  the  Applique  map  and  data  update  cycle  for  adequate 
vehicle  navigation. 

CSS  Applique 

-  Improve  reliability  of  communication  links  and  simplify  complex  input  screens. 

-  Increase  situation  awareness  by  allowing  Applique  to  access  multiple  radio  nets. 

-  Improve  ability  to  request  support  using  Applique  by  providing  an 
acknowledgment  of  request  from  the  receiving  unit. 

-  Increase  situational  awareness  by  improving  Applique  screen  resolution. 

-  Simplify  the  cryptic  and  unusable  addressing  system. 

-  Continue  to  develop  and  mature  the  Applique  CSS  functions. 

Fire  Support  Applique 

-  Continue  development  of  the  applique  while  improving  connectivity  problems. 

-  Integrate  Applique  with  AFATDS. 
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M/CM/S  Applique 


-  The  Applique  capabilities  should  be  matured  to  provide  the  Engineers  with  a 
robust  method  to  produce  graphics  and  overlays  for  identifying  friendly  and 
enemy  obstacles. 

-  The  tactical  internet  architecture  for  the  engineer  force  structure  should  be 
modified  to  meet  mission  requirements.  The  architecture  will  need  changing  so  it 
provides  the  capability  to  communicate  within  the  supported  maneuver  force  as 
well  as  within  the  engineer  battalion. 

-  The  practice  of  disseminating  known  obstacle  information  and  receiving  template 
obstacle  locations  on  MCS/P  and  Applique  overlays  should  be  enhanced. 

-  The  graphics  for  minefields  need  to  be  military  graphics,  not  just  a  box  on  the 
screen  and  symbols  need  to  be  drag  and  drop. 
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APPENDIX  B.  SURVEY  COVER  LETTER 


134  Brownell  Circle 
Monterey,  CA  93940 
831-372-8150 
kwstraye@nps.navy.mil 


Dear 


I  am  an  Army  student  in  the  Acquisition  Management  program  at  the  Naval 
Postgraduate  School  (NPS),  Monterey,  California.  Thank  you  for  agreeing  to  help  my 
research  on  the  use  of  Advanced  Warfighting  Experiments  (AWE).  By  completing  the 
enclosed  survey,  you  are  providing  valuable  insight  into  the  results  of  the  1997  Army 
Task  Force  XXI  AWE. 

My  research  will  evaluate  the  effectiveness  of  the  1997  AWE  objective  of 
providing  information  to  support  investment  decisions  and  refinement  of  requirements  for 
information  age  technologies.  The  enclosed  survey  collects  data  from  involved  program 
offices  and  user  representatives  to  determine  the  perceived  utility  of  the  AWE  data. 
Opinions  detailing  why  specific  AWE  recommendations  were  or  were  not  implemented 
will  assist  in  determining  the  contributing  factors  to  a  program’s  ability  to  benefit  from 
participation  in  the  experiment. 

My  research  will  make  recommendations  on  how  to  best  use  Advanced 
Warfighting  Experiments  to  support  acquisition  decisions.  This  study  will  assist 
acquisition  managers  in  gaining  maximum  benefit  from  participation  in  future 
experiments.  Also,  the  research  will  assist  planners  in  tailoring  future  experiments  to 
better  benefit  participating  progr^s. 

The  survey  includes  four  sections: 

■  Section  one  collects  information  about  the  specific  level  of  implementation  of 
recommendations  made  for  your  program  by  the  Army  Operational  Test  and 
Evaluation  Command’s  Live  Experiment  Assessment  Report. 

■  Section  two  contains  questions  about  your  program’s  experience  in  the  AWE. 

■  Section  three  asks  you  to  rate  given  reasons  why  AWE  recommendations  may  or 
may  not  have  been  fully  implemented  in  your  program. 

■  Section  four  is  available  for  you  to  provide  any  other  information  that  you  think 
may  be  valuable  to  my  study. 


115 


For  your  convenience,  I  have  provided  this  survey  as  an  attached  word  document. 
You  may  email  the  completed  survey  to  me  or  mail  a  printed  copy  to  the  address  above. 
For  your  convenience,  I  will  send  a  copy  of  the  survey  by  mail  to  you  in  several  weeks 
with  a  stamped,  self-addressed  envelope.  I  estimate  that  the  survey  should  take  between 
thirty  and  forty-five  minutes  to  review  and  make  comments.  I  would  be  happy  to  send 
you  an  electronic  copy  of  my  final  effort  for  your  own  use. 

If  you  need  any  additional  details,  please  contact  me  at  the  above  address,  email, 
or  phone.  All  information  gathered  will  be  used  on  a  non-attribution  basis  and  the  names 
of  individuals  providing  opinions  will  be  kept  in  strict  confidence.  Names  of  programs 
will  be  kept  confidential  upon  request.  Please  feel  free  to  distribute  this  survey  to  others 
that  may  have  experience  relating  to  my  research.  I  appreciate  your  help  on  this  project. 

Sincerely, 


Kenneth  W.  Strayer 
Captain,  United  States  Army 
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